How to return a csv file/Pandas DataFrame in JSON format using FastAPI?

ghz 1years ago ⋅ 4892 views

Question

I have a .csv file that I would like to render in a FastAPI app. I only managed to render the .csv file in JSON format as follows:

def transform_question_format(csv_file_name):

    json_file_name = f"{csv_file_name[:-4]}.json"

    # transforms the csv file into json file
    pd.read_csv(csv_file_name ,sep=",").to_json(json_file_name)

    with open(json_file_name, "r") as f:
        json_data = json.load(f)

    return json_data

@app.get("/questions")
def load_questions():

    question_json = transform_question_format(question_csv_filename)

    return question_json

When I tried returning directly pd.read_csv(csv_file_name ,sep=",").to_json(json_file_name), it works, as it returns a string.

How should I proceed? I believe this is not the good way to do it.

Answer

The below shows four different ways of returning the data stored in a .csv file/Pandas DataFrame (for solutions without using Pandas DataFrame, have a look here). Related answers on how to efficiently return a large dataframe can be found here and here as well.

Option 1

The first option is to convert the file data into JSON and then parse it into a dict. You can optionally change the orientation of the data using the orient parameter in the .to_json() method.

Note: Better not to use this option. See Updates below.

from fastapi import FastAPI
import pandas as pd
import json

app = FastAPI()
df = pd.read_csv("file.csv")

def parse_csv(df):
    res = df.to_json(orient="records")
    parsed = json.loads(res)
    return parsed
    
@app.get("/questions")
def load_questions():
    return parse_csv(df)

  • Update 1 : Using .to_dict() method would be a better option, as it would return a dict directly, instead of converting the DataFrame into JSON (using df.to_json()) and then that JSON string into dict (using json.loads()), as described earlier. Example:

    @app.get("/questions")

    def load_questions(): return df.to_dict(orient="records")

  • Update 2 : When using .to_dict() method and returning the dict, FastAPI, behind the scenes, automatically converts that return value into JSON using the Python standard json.dumps(), after converting it into JSON-compatible data first, using the jsonable_encoder, and then putting that JSON-compatible data inside of a JSONResponse (see this answer for more details). Thus, to avoid that extra processing, you could still use the .to_json() method, but this time, put the JSON string in a custom Response and return it directly, as shown below:

    from fastapi import Response

    @app.get("/questions") def load_questions(): return Response(df.to_json(orient="records"), media_type="application/json")

Option 2

Another option is to return the data in string format, using .to_string() method.

@app.get("/questions")
def load_questions():
    return df.to_string()

Option 3

You could also return the data as an HTML table, using .to_html() method.

from fastapi.responses import HTMLResponse

@app.get("/questions")
def load_questions():
    return HTMLResponse(content=df.to_html(), status_code=200)

Option 4

Finally, you can always return the file as is using FastAPI's [FileResponse](https://fastapi.tiangolo.com/advanced/custom- response/#fileresponse).

from fastapi.responses import FileResponse

@app.get("/questions")
def load_questions():
    return FileResponse(path="file.csv", filename="file.csv")
#fastapi