tqdm Python Library

Group:J017

Members:

Boinapally Dhanvini-24110079
Bollineni Snehitha Rao-24110081
Paleti Nikhitha Chowdary-24110233

Introduction to python library:tqdm:

tqdm is a Python package that produces progress bars for loops. It is meant to be fast, versatile, and convenient. Its name is based on the Arabic word taqadum (meaning “progress”) and is also an abbreviation for te quiero demasiado (“I love you so much” in Spanish).

When executing loops in programming, it’s frequently useful to know how far the process has proceeded. One technique to accomplish this is to print the number of the current loop, but it has some issues: - There can be too many print statements, which slow things down. - It does not indicate the speed of the loop. - There is no indication of the remaining time. - The data isn’t neatly presented in one location.

Correcting them manually requires additional effort and might not be suited for other scenarios. tqdm addresses all these issues by giving the flexibility to easily attach a tidy, configurable progress bar to loops without slowing them down.

tqdm can be applied in a wide range of areas: - It makes users observe the processing of tasks such as data processing or file download. - Developers can utilize it for debugging and error tracking, for example, during machine learning training. - It’s an excellent tool to use when teaching Python. - The command-line interface (CLI) version is useful for system administrators who want to track data flow in scripts.

With tqdm, it is easy and fast to add progress bars, and coding becomes more efficient and user-friendly.

Installation of tqdm python library:

pip install tqdm

Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: tqdm in c:\users\dell\appdata\roaming\python\python312\site-packages (4.67.1)
Requirement already satisfied: colorama in c:\users\dell\appdata\roaming\python\python312\site-packages (from tqdm) (0.4.6)
Note: you may need to restart the kernel to use updated packages.


[notice] A new release of pip is available: 24.3.1 -> 25.0.1
[notice] To update, run: python.exe -m pip install --upgrade pip

To create a progress bar, we use the tqdm() function from the tqdm module to wrap our iterable. Here’s a simple example: the time.sleep(0.01) function acts as a placeholder for the actual code that would run during each iteration.

from tqdm import tqdm
import time
for i in tqdm(range(1000)):
    time.sleep(0.01)

100%|██████████| 1000/1000 [00:11<00:00, 88.50it/s]

KEY FEATURES OF TQDM AND IT’S USECASES (With few examples):

1.Basic Loop Progress Bar:

This feature in tqdm tracks the loop execution and gives a progress bar which :

✔ Shows percentage completed

✔ Estimates remaining time

✔ Displays iterations per second(it/s)

It is also called as Python Iterable Wrapper as it wraps around python iterables.

What Does Each Iteration Do?

tqdm(range(10)) → Tqdm wraps range(10), so the progress of the loop is tracked by tqdm.

time.sleep(1) → Introduces a 1-second pause per iteration (emulating a long-running process).

tqdm updates the progress bar in real time, displaying:

1)Iterations done / Total iterations

2)Elapsed time

3)Iterations per second

#Example
from tqdm import tqdm
import time

for i in tqdm(range(10)):
    time.sleep(1)

100%|██████████| 10/10 [00:10<00:00,  1.01s/it]

These is the outputs while the code is running:

2.Works with map():

Tqdm can monitor the execution of map() functions:

tqdm can monitor the execution of map() functions, which makes it helpful for parallel processing, data transformation, and big computations.

Explanation for below code:

✔ map(process, data): Calls process(n) for each value in data.

✔ tqdm(., total=len(data)): Wraps map() to show a progress bar.

✔ desc=“Processing Data”: Adds a custom label to the bar.

from tqdm import tqdm
import time

def process(x):
    time.sleep(0.5)
    return x * x

results = list(tqdm(map(process, range(10)), total=10, desc="Processing"))

Processing: 100%|██████████| 10/10 [00:05<00:00,  1.98it/s]

These are the outputs when the code is running:

3.Pandas Support:

Tqdm integrates seamlessly with Pandas, adding progress bars to progress_apply():

tqdm is well-supported with Pandas and will include a progress bar when using .progress_apply(). This would be helpful for large datasets for data processing, feature engineering, and data cleaning operations.

Explanation for below code:

✔ tqdm.pandas() → Makes tqdm progress bars available for pandas operations.

✔ df[‘values’].progress_apply(process) → Displays a progress bar as process(x) is applied to every row.

pip install pandas

Defaulting to user installation because normal site-packages is not writeableNote: you may need to restart the kernel to use updated packages.


[notice] A new release of pip is available: 24.3.1 -> 25.0.1
[notice] To update, run: python.exe -m pip install --upgrade pip


Requirement already satisfied: pandas in c:\users\dell\appdata\roaming\python\python312\site-packages (2.2.3)
Requirement already satisfied: numpy>=1.26.0 in c:\users\dell\appdata\roaming\python\python312\site-packages (from pandas) (2.2.1)
Requirement already satisfied: python-dateutil>=2.8.2 in c:\users\dell\appdata\roaming\python\python312\site-packages (from pandas) (2.9.0.post0)
Requirement already satisfied: pytz>=2020.1 in c:\users\dell\appdata\roaming\python\python312\site-packages (from pandas) (2025.1)
Requirement already satisfied: tzdata>=2022.7 in c:\users\dell\appdata\roaming\python\python312\site-packages (from pandas) (2025.1)
Requirement already satisfied: six>=1.5 in c:\users\dell\appdata\roaming\python\python312\site-packages (from python-dateutil>=2.8.2->pandas) (1.17.0)


import pandas as pd
from tqdm import tqdm
import time



tqdm.pandas()
df = pd.DataFrame({'A': range(1000)})
df['B'] = df['A'].progress_apply(lambda x: (time.sleep(0.01), x**2)[1])

100%|██████████| 1000/1000 [00:11<00:00, 88.67it/s]

The changes in otput while the code is running:

#Without using time.sleep()
import pandas as pd
from tqdm import tqdm

tqdm.pandas()
df = pd.DataFrame({'A': range(1000)})
df['B'] = df['A'].progress_apply(lambda x: x**2)
#Without time.sleep(0.01), the operation runs extremely fast since squaring a number is quick resulting in a high iteration speed (e.g., 143,512 it/s).
#Adding time.sleep(0.01) forces a delay in each iteration, slowing down execution to around 97 it/s, making the progress bar update much slower.

100%|██████████| 1000/1000 [00:00<00:00, 176416.57it/s]

4.Nested Bars:

Tqdm can track inner and outer loops simultaneously:

Explanation:

✔ tqdm(range(outer_loops), desc=“Outer Loop”) → Monitors the outer loop.

✔ tqdm(range(inner_loops), desc=“Inner Loop”, leave=False) → Monitors the inner loop and suppresses it after it is done.

✔ leave=False → Avoids clutter by suppressing completed inner loop progress bars.

from tqdm.notebook import tqdm  # Use notebook version
import time

for i in tqdm(range(3), desc="Outer Loop", leave=True):
    for j in tqdm(range(5), desc="Inner Loop", leave=True):
        time.sleep(0.5)

These are the outputs when the code is running:

from tqdm import tqdm # Import the tqdm function from the tqdm library
import time

for i in tqdm(range(3), desc="Outer Loop"):
    for j in tqdm(range(5), desc="Inner Loop", leave=False):
        time.sleep(0.5)  # Simulating work

Outer Loop: 100%|██████████| 3/3 [00:07<00:00,  2.59s/it]

These are the outputs while the code is running:

from tqdm.notebook import tqdm  # Use notebook version
import time

for i in tqdm(range(3), desc="Outer Loop", leave=True):
    for j in tqdm(range(5), desc="Inner Loop", leave=False):
        time.sleep(0.5)

These are the outputs while the code is running:

from tqdm.notebook import tqdm  # Use notebook version
import time

for i in tqdm(range(3), desc="Outer Loop", leave=False):
    for j in tqdm(range(5), desc="Inner Loop", leave=False):
        time.sleep(0.5)
#With leave=True (default), progress bars remain visible after completion, useful for tracking all iterations.
#With leave=False, the progress bar disappears once the loop finishes

The code output while it is running are same as the above code, but there is no utput seen.

5.Customization:

tqdm provides multiple customization options to enhance the progress bar’ appearance, format, and behavior. Here are some key ways to customize it:

Explanation:

{desc} Displays the custom description (desc=” Processing Data”)

{percentage:3.0f}% Shows progress percentage with 3-character padding (3.0f ensures no decimal places)

{bar} The progress bar itself

{n_fmt} Current iteration count (formatted for readability)

{total_fmt} Total iterations (formatted for readability)

” iters” Static text added at the end

from tqdm.notebook import tqdm
import time

# Customizing tqdm progress bar
for i in tqdm(
    range(10),
    desc=" Processing Data",  # Custom description
    colour="green",  # Change progress bar color
    ncols=500,  # Set custom width
    bar_format="{desc}: {percentage:3.0f}%|{bar}| {n_fmt}/{total_fmt} iters",  # Custom format
    leave=True,  # Remove completed bars
):
    time.sleep(0.3)  # Simulating a delay

These are the outputs while the above code is running:

6.tqdm with Lists and Enumerate:

Track progress when iterating over a list.

my_list = ["Task 1", "Task 2", "Task 3"]
for task in tqdm(my_list, desc="Processing Tasks"):
    time.sleep(0.5)  # Simulating processing time

These are the outputs while the above code is running:

Explanation:

✔ tqdm(my_list, desc=“Processing tasks”) → Wraps the list with a progress bar.

✔ time.sleep(0.5) → Simulates a slow operation.

Monitors list processing in real-time

for index, item in tqdm(enumerate(my_list), total=len(my_list), desc="Processing Tasks"):
    time.sleep(0.5)  # Simulating work

These are the outputs for the bove code while it is running:

Explanation:

✔ enumerate(my_list) → Returns value and index while iterating.

✔ total=len(my_list) → Maintains the progress bar correctly updated.

✔ desc=“Processing Tasks” → Includes a personalized description.

Monitors both index and value processing

7.File Downloads:

Tqdm can monitor the progress of large downloads:

tqdm is able to monitor the progress of downloading large files, displaying a live progress bar during download. This comes in handy when downloading large data sets, media, or software updates, allowing you to keep track of the speed and time to complete.

import requests  # Handles HTTP requests
from tqdm import tqdm  # Displays a progress bar

# Define the file URL
url = "https://example.com/large_file.zip"

# Initiate the HTTP request with streaming enabled
response = requests.get(url, stream=True)

# Extract total file size from headers (returns 0 if not available)
file_size = int(response.headers.get('content-length', 0))

# Open the file for writing in binary mode while using tqdm to track progress
with open("file.zip", "wb") as file, tqdm(
    desc="Downloading",  # Custom description for the progress bar
    total=file_size,  # Total number of bytes expected
    unit="B",  # Display progress in Bytes
    unit_scale=True,  # Automatically adjust units (KB, MB, GB)
    unit_divisor=1024,  # Use 1024 for binary conversion (instead of 1000)
) as bar:

    # Iterate over the response content in chunks of 1 KB
    for chunk in response.iter_content(chunk_size=1024):
        file.write(chunk)  # Write the chunk to file
        bar.update(len(chunk))  # Increment progress bar by the chunk size

Downloading: 100%|██████████| 1.23k/1.23k [00:00<00:00, 1.21MB/s]

The above code has no sleep, so no time delay, the final output is directly shown as above.

Explanation:

✔ requests.get(url, stream=True) → Downloads the file in pieces rather than downloading it all at once.

✔ response.headers.get(‘content-length’,0) → Obtains the file size to monitor progress.

✔ tqdm(., total=file_size, unit=“B”, unit_scale=True) → Displays download progress in bytes (B), KB, MB, or GB.

✔ response.iter_content(chunk_size=block_size) → Reads the file in small pieces (1KB) to conserve memory.

✔ bar.update(len(chunk)) → Advances the progress bar with each piece.

8.Command-Line Interface (CLI) in tqdm:

A Command-Line Interface (CLI) is when the user uses a terminal to give commands to a program, as opposed to the graphical user interface. tqdm comes with a CLI mode to follow progress through terminal commands. In the command line, a pipe (|) is how to take output from one command and input to another.

1.count lines of text in all *.txt files:

code: $ cat *.txt | wc -l

Output: 1075075

2.same as above same but with continuously updating progress information:

code:$ cat *.txt | python3 -m tqdm –unit loc –unit_scale | wc -l

Output:1.08Mloc [00:07, 142kloc/s]

3.same if total is known:

code: $ cat *.txt | python3 -m tqdm –unit loc –unit_scale –total 1075075 | wc -l

output:

100%|#####################################| 1.08/1.08M [00:07<00:00, 142kloc/s]

1075075

9.tqdm with Parallel Processing (Threading & Multiprocessing):

Parallel processing can be used to accelerate execution by executing multiple tasks simultaneously. Progress tracing in parallel execution, however, can be problematic. tqdm is compatible with Threading and Multiprocessing for visualizing progress in concurrent tasks. Threading is beneficial for I/O-bound tasks (downloading files, web scraping, etc.). Multiprocessing is better for CPU-bound tasks (e.g., data processing, machine learning).

from tqdm import tqdm
from concurrent.futures import ThreadPoolExecutor
import time

def task(n):
    time.sleep(0.5)  # Simulate work
    return n * n

with ThreadPoolExecutor(max_workers=4) as executor:
    results = list(tqdm(executor.map(task, range(10)), total=10, desc="Parallel Processing"))

Parallel Processing: 100%|██████████| 10/10 [00:01<00:00,  6.71it/s]

The output changes while the code runs:

Explanation for the above code:

1)Function task(n): Simulates a long-running operation with time.sleep(0.5).

2)Returns n n (square of n). Using ThreadPoolExecutor(max_workers=4)*:

Initializes a thread pool of 4 worker threads, and tasks can run in parallel. Each worker thread takes a number from range(10) and executes task(n).

3)Using executor.map(task, range(10)):

Maps the task(n) function to numbers 0-9 in parallel using 4 threads. If max_workers=4, a maximum of 4 numbers are processed simultaneously.

4)Using tqdm to Track Progress:

tqdm(executor.map(.), total=10, desc=“Parallel Processing”) Shows a progress bar while the tasks finish.

import pandas as pd
from tqdm import tqdm
from concurrent.futures import ProcessPoolExecutor
import time

The outputs while the code runs:

Explanation:

1)Function compute(n): Simulates a CPU-intensive task with time.sleep(0.5) and Returns n * n (square of n).

2)Using ProcessPoolExecutor(max_workers=4):

Creates a pool of 4 processes to execute tasks in parallel and each process picks a number from range(10) and applies compute(n). Unlike threads, each process runs in its own memory space, avoiding GIL (Global Interpreter Lock) bottlenecks.

3)Using executor.map(compute, range(10)):

Maps the compute(n) function to numbers 0-9 concurrently using 4 processes. If max_workers=4, up to 4 numbers are processed at the same time.

4)Using tqdm to Track Progress:

tqdm(executor.map(…), total=10, desc=“Multiprocessing Example”) Displays a progress bar while tasks complete.

EXAMPLES OF TQDM

1. Comparing Loop Execution(Without and with tqdm):

Without tqdm(No progress feedback)

import time

for i in range(10):
    time.sleep(0.5)
print("Task Done!")

Task Done!

Expected Output:

After 5 Seconds, it gives the message:

Task Done!

With tqdm(Real-time tracking)

from tqdm import tqdm
import time

for i in tqdm(range(10), desc="Processing"):
    time.sleep(0.5)
print("Task Done!")

Processing: 100%|██████████| 10/10 [00:05<00:00,  2.00it/s]

Task Done!

Expected Output:

A real-time progress bar appears in the cell output, which update for every 0.5 seconds:

Processing:  10%|██          | 1/10 [00:00<00:04,  2.00s/it]
Processing:  20%|████        | 2/10 [00:01<00:04,  2.00s/it]
Processing:  30%|██████      | 3/10 [00:01<00:03,  2.00s/it]
...
Processing: 100%|██████████| 10/10 [00:05<00:00,  2.00s/it]
Task Done!

✔ Progress bar updates in real-time.

2. File Downlaod Progress:

import requests
from tqdm import tqdm
import time
url = "https://example.com/largefile.zip"
response = requests.get(url, stream=True)

total_size = int(response.headers.get('content-length', 0))
block_size = 1024

with open("downloaded_file.zip", "wb") as file, tqdm(
    total=total_size, unit="B", unit_scale=True, desc="Downloading"
) as progress_bar:
    for data in response.iter_content(block_size):
        file.write(data)
        progress_bar.update(len(data))

Downloading: 100%|██████████| 1.26k/1.26k [00:00<00:00, 1.91MB/s]

Expected Output(Example of a 10MB File):

Downloading:  10%|███        | 1.0M/10M [00:01<00:09,  1.0MB/s]
Downloading:  20%|█████      | 2.0M/10M [00:02<00:08,  1.0MB/s]
...
Downloading: 100%|██████████| 10M/10M [00:10<00:00,  1.0MB/s]

✔ Shows real-time file download progress.

3. Processing a Large Dataset(pandas):

import pandas as pd
from tqdm import tqdm

tqdm.pandas(desc="Processing Data")

df = pd.DataFrame({'col1': range(1000)})
df['col2'] = df['col1'].progress_apply(lambda x: x * 2)

Processing Data: 100%|██████████| 1000/1000 [00:00<00:00, 119720.96it/s]

Expected Output:

Processing Data: 100%|██████████| 1000/1000 [00:00<00:00, 119720.96it/s]

✔ Shows how many rows have been processed.

4. Multi-threading with tqdm:

from tqdm import tqdm
import time
from concurrent.futures import ThreadPoolExecutor

def task(n):
    time.sleep(n)
    return n

with ThreadPoolExecutor(max_workers=3) as executor:
    results = list(tqdm(executor.map(task, [2, 3, 1]), total=3, desc="Threads"))

Threads: 100%|██████████| 3/3 [00:02<00:00,  1.00it/s]

Expected Output:

Threads: 100%|██████████| 3/3 [00:03<00:00,  1.00s/it]

✔ Tracks the progress of multi-threaded tasks.

5. Nested Progress Bars:

from tqdm import trange
import time

for i in trange(3, desc="Outer Loop"):
    for j in trange(5, desc="Inner Loop", leave=False):
        time.sleep(0.2)

Outer Loop:   0%|          | 0/3 [00:00<?, ?it/s]
Inner Loop:   0%|          | 0/5 [00:00<?, ?it/s]
Inner Loop:  20%|██        | 1/5 [00:00<00:00,  5.00it/s]
Inner Loop:  40%|████      | 2/5 [00:00<00:00,  4.97it/s]
Inner Loop:  60%|██████    | 3/5 [00:00<00:00,  4.95it/s]
Inner Loop:  80%|████████  | 4/5 [00:00<00:00,  4.94it/s]
Inner Loop: 100%|██████████| 5/5 [00:01<00:00,  4.94it/s]
Outer Loop:  33%|███▎      | 1/3 [00:01<00:02,  1.02s/it]
Inner Loop:   0%|          | 0/5 [00:00<?, ?it/s]
Inner Loop:  20%|██        | 1/5 [00:00<00:00,  5.00it/s]
Inner Loop:  40%|████      | 2/5 [00:00<00:00,  4.96it/s]
Inner Loop:  60%|██████    | 3/5 [00:00<00:00,  4.95it/s]
Inner Loop:  80%|████████  | 4/5 [00:00<00:00,  4.94it/s]
Inner Loop: 100%|██████████| 5/5 [00:01<00:00,  4.94it/s]
Outer Loop:  67%|██████▋   | 2/3 [00:02<00:01,  1.02s/it]
Inner Loop:   0%|          | 0/5 [00:00<?, ?it/s]
Inner Loop:  20%|██        | 1/5 [00:00<00:00,  5.00it/s]
Inner Loop:  40%|████      | 2/5 [00:00<00:00,  4.96it/s]
Inner Loop:  60%|██████    | 3/5 [00:00<00:00,  4.95it/s]
Inner Loop:  80%|████████  | 4/5 [00:00<00:00,  4.94it/s]
Inner Loop: 100%|██████████| 5/5 [00:01<00:00,  4.94it/s]
Outer Loop: 100%|██████████| 3/3 [00:03<00:00,  1.02s/it]

Expected Output:

Outer Loop:  33%|█████    | 1/3 [00:01<00:02,  1.00s/it]
Inner Loop:  20%|██        | 1/5 [00:00<00:01,  2.00s/it]
Inner Loop:  40%|████      | 2/5 [00:00<00:01,  2.00s/it]
Inner Loop: 100%|██████████| 5/5 [00:01<00:00,  2.00s/it]
Outer Loop:  67%|████████  | 2/3 [00:02<00:01,  1.00s/it]
...
Outer Loop: 100%|██████████| 3/3 [00:03<00:00,  1.00s/it]

✔ Shows real-time file download progress.

Common Tqdm Issues and Solutions :

Progress Bar Not Updating A common issue with tqdm is the progress bar not updating, often due to output buffering, especially in Jupyter Notebooks. To fix this, use tqdm.notebook, which provides a GUI-based progress bar with color-coded status indicators: blue (normal), green (completed), and red (interrupted/error).

Flushing the Output Stream Another solution is explicitly flushing the output stream to ensure real-time updates. By using sys.stdout.flush(), you force the buffered output to display immediately, preventing delays. However, frequent flushing may introduce performance overhead.

import sys
import time
from tqdm import tqdm

with tqdm(total=50) as progress_bar:
    for _ in range(50):
        time.sleep(0.05)  # Simulating work
        progress_bar.update(1)
        sys.stdout.flush()  # Force immediate output update

100%|██████████| 50/50 [00:02<00:00, 19.52it/s]

COMPATABILITY ISSUES:

Tqdm is compatible in most Python environments, although there might be incompatibilities with custom output streams, third-party libraries, or version mismatches. Redirection of output to files or with unsupported streams will hinder correct display, while certain libraries can affect tqdm’s functionality. Version compatibility and the correction of any needed adjustments can resolve these conflicts.

-Utilize supported output streams to guarantee the progress bar is correctly displayed.

-Verify conflicts with third-party libraries and adjust settings if necessary.

-Upgrade, downgrade, or request community support if compatibility issues continue.

CONCLUSION

TQDM is a light yet powerful Python library that adds progress bars to loops, downloads, and data processing with very little code modifications.
It’s light, fast, and highly flexible, with support for nested loops, multi-threading, and pandas.
It also facilitates debugging and monitoring, making it a must-have.
The one limitation is that it may not be ideal for extremely fast loops, but its strengths greatly outshine this slight limitation.
In programming, knowing how long something will take can be a lifesaver. Instead of sitting in the dark, tqdm provides you with accurate progress reports, making running more transparent and user-friendly.With tqdm, we don’t need to wait blindly,we know exactly how long to wait.

REFERENCES:

https://www.theoj.org/joss-papers/joss.01277/10.21105.joss.01277.pdf

2)https://www.geeksforgeeks.org/python-how-to-make-a-terminal-progress-bar-using-tqdm/

https://tqdm.github.io/
https://www.datacamp.com/tutorial/tqdm-python