panda df not showing all rows after loading from MS SQL

2024/9/22 10:37:38

I'm using Pandas with latest sqlalchemy (1.4.36) to query a MS SQL DB, using the following Python 3.10.3 [Win] snippet:

import pandas as pd                                             # 
from sqlalchemy import create_engine, event
from sqlalchemy.engine.url import URL
# ...
def get_table_columns():SQLA = 'SELECT TABLE_NAME,COLUMN_NAME FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_NAME LIKE \'pa%\' ORDER BY TABLE_NAME;' # Use pandas for getting named table & columnsconn_str = set_db_info()conn_url = URL.create("mssql+pyodbc", query={"odbc_connect": conn_str})engine = create_engine(conn_url)df = pd.read_sql(SQLA, engine)# Permanently changes the pandas settingspd.set_option('display.max_rows', None)pd.set_option('display.max_columns', None)print(df)return df

However, this only prints the first 292 rows, and not all of the entire 2351 rows. Using REPL, I can check this with:

>>> z = get_table_columns()
>>> zTABLE_NAME        COLUMN_NAME
0                    paacc           accesscd
... # <-- I added these
292                paapepi        piapeheadat>>> z.count()
TABLE_NAME     2351
COLUMN_NAME    2351
dtype: int64>>> z.shape[0]
2351>>> z.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 2351 entries, 0 to 2350
Data columns (total 2 columns):#   Column       Non-Null Count  Dtype
---  ------       --------------  -----0   TABLE_NAME   2351 non-null   object1   COLUMN_NAME  2351 non-null   object
dtypes: object(2)
memory usage: 36.9+ KB

Q: What is going on, and why can't I print/show all the rows?

Answer

To display all the rows in pandas, you should set the display option to None or 1 extra from the dataframe size as you have done in your code:

pd.set_option('display.max_rows', None)
pandas.set_option('display.max_rows', z.shape[0]+1)

Given that this is not the problem, it may be that the IDE or program that you use automatically crops this information from the view (For example Ipython crops every big output).

Other thing to try is to force the print of the dataframe instead of just is return value:

>>> print(z)

To inspect everything I would recommend you to pass that into a csv/excel file to do it better.

https://en.xdnf.cn/q/119144.html

Related Q&A

How to extract multiple grandchildren/children from XML where one child is a specific value?

Im working with an XML file that stores all "versions" of the chatbot we create. Currently we have 18 versions and I only care about the most recent one. Im trying to find a way to extract a…

Is there are a way to replace python import with actual sources?

I am having python files with the import statements which I would like to replace into the actual code placed in the foo.py.For instance, with the in file:from foo import Barbar = Bar() print barI woul…

How to use supported numpy and math functions with CUDA in Python?

According to numba 0.51.2 documentation, CUDA Python supports several math functions. However, it doesnt work in the following kernel function: @cuda.jit def find_angle(angles):i, j = cuda.grid(2)if i …

PYTHON - Remove tuple from list if contained in another list

I have a list of tuples:list_of_tuples = [(4, 35.26), (1, 48.19), (5, 90.0), (3, 90.0)]tuple[0] is an item_IDtuple[1] is an angleI have a list of item_IDs I want to remove/ignore from the list:ignore_I…

How to run a ij loop in Python, and not repeat (j,i) if (i,j) has already been done?

I am trying to implement an "i not equal to j" (i<j) loop, which skips cases where i = j, but I would further like to make the additional requirement that the loop does not repeat the perm…

Split a string with multiple delimiters

I have the string "open this and close that" and I want to obtain "open this and" and "close that". This is my best attempt:>>>print( re.split(r[ ](?=(open|clos…

Extracting a string between 2 chracters using python [duplicate]

This question already has answers here:Python-get string between to characters(4 answers)Closed 7 years ago.I need a Python regex to give me all the strings between ~ and ^ from a string like this:~~~~…

remove empty line printed from hive query output using python

i am performing a hive query and storing the output in a tsv file in the local FS. I am running a for loop for the hive query and passing different parameters. If the hive query returns no output once …

.exceptions.WebDriverException: Message: Can not connect to the Service

struggling to find a solution all over, have latest chrome 117 and also downloaded chromedriver and used the path accordingly in script also tried with chrome browser Although it opens the browser but …

How to call a previous function in a new function? [closed]

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.This question does not appear to be about programming within the scope defined in the help center.Cl…