CSVtoJSON

This project is not dependent on others packages or libraries, and supports both synchronous and Promise-based asynchronous APIs.

This repository uses

Follow me, and consider starring the project to show your :heart: and support.

Table of Contents

• Description
• Support for JS & TS
• Prerequisites
• Install npm convert-csv-to-json package
  • Install
  • Sync API Usage
    • Generate JSON file
    • Generate Array of Object in JSON format
    • Generate Object with sub array
    • Define field delimiter
    • Trim header field
    • Trim header field with whitespaces
    • Support Quoted Fields
    • Index header
    • Empty rows
    • Format property value by type
      • Number
      • Boolean
    • Encoding
    • Working with CSV strings directly
  • Async API Usage
    • Basic Async Operations
    • Working with Raw CSV Data
    • Processing Large Files
    • Error Handling and Retries
    • Batch Processing
  • Chaining Pattern
• Development
• CI CD github action
• License
• Buy me a Coffee

Description

Converts csv files to JSON files with Node.js. Supports both synchronous operations and Promise-based asynchronous operations, allowing integration with modern async/await patterns.

Give an input file like:

first_name	last_name	email	gender	age	zip	registered
Constantin	Langsdon	clangsdon0@hc360.com	Male	96	123	true
Norah	Raison	nraison1@wired.com	Female	32		false

e.g. :

first_name;last_name;email;gender;age;zip;registered
Constantin;Langsdon;clangsdon0@hc360.com;Male;96;123;true
Norah;Raison;nraison1@wired.com;Female;32;;false

will generate:

[
 {
  "first_name": "Constantin",
  "last_name": "Langsdon",
  "email": "clangsdon0@hc360.com",
  "gender": "Male",
  "age": "96",
  "zip": "123",
  "registered": "true"
 },
 {
  "first_name": "Norah",
  "last_name": "Raison",
  "email": "nraison1@wired.com",
  "gender": "Female",
  "age": "32",
  "zip": "",
  "registered": "false"
 }
]

Support for JS & TS

This package is compatible with and .

Prerequisites

NPM (see Installing Npm).

Install npm convert-csv-to-json package

Go to NPM package convert-csv-to-json.

Install

Install package in your package.json

$ npm install convert-csv-to-json --save

Install package on your machine

$ npm install -g convert-csv-to-json

Sync API Usage

Generate JSON file

let csvToJson = require('convert-csv-to-json');

let fileInputName = 'myInputFile.csv'; 
let fileOutputName = 'myOutputFile.json';

csvToJson.generateJsonFileFromCsv(fileInputName,fileOutputName);

Generate Array of Object in JSON format

let csvToJson = require('convert-csv-to-json');

let json = csvToJson.getJsonFromCsv("myInputFile.csv");
for(let i=0; i<json.length;i++){
    console.log(json[i]);
}

Generate Object with sub array

firstName;lastName;email;gender;age;birth;sons
Constantin;Langsdon;clangsdon0@hc360.com;Male;96;10.02.1965;*diego,marek,dries*

Given the above CSV example, to generate a JSON Object with properties that contains sub Array, like the property sons with the values diego,marek,dries you have to call the function parseSubArray(delimiter, separator) . To generate the JSON Object with sub array from the above CSV example:

    csvToJson.parseSubArray('*',',')
                .getJsonFromCsv('myInputFile.csv');

The result will be:

[
  {
      "firstName": "Constantin",
      "lastName": "Langsdon",
      "email": "clangsdon0@hc360.com",
      "gender": "Male",
      "age": "96",
      "birth": "10.02.1965",
      "sons": ["diego","marek","dries"]
    }
]

Define field delimiter

A field delimiter is needed to split the parsed values. As default the field delimiter is the semicolon (;), this means that during the parsing when a semicolon (;) is matched a new JSON entry is created. In case your CSV file has defined another field delimiter you have to call the function fieldDelimiter(myDelimiter) and pass it as parameter the field delimiter.

E.g. if your field delimiter is the comma , then:

 csvToJson.fieldDelimiter(',')
            .getJsonFromCsv(fileInputName);

Trim header field

The content of the field header is cut off at the beginning and end of the string. E.g. “ Last Name “ -> “Last Name”.

Trim header field with whitespaces

Use the method trimHeaderFieldWhiteSpace(true) to remove the whitespaces in an header field (E.g. “ Last Name “ -> “LastName”):

 csvToJson.trimHeaderFieldWhiteSpace(true)
            .getJsonFromCsv(fileInputName);

Support Quoted Fields

To be able to parse correctly fields wrapped in quote, like the last_name in the first row in the following example:

first_name	last_name	email
Constantin	“Langsdon,Nandson,Gangson”	clangsdon0@hc360.com

you need to activate the support quoted fields feature:

 csvToJson.supportQuotedField(true)
            .getJsonFromCsv(fileInputName);

The result will be:

[
  {
      "firstName": "Constantin",
      "lastName": "Langsdon,Nandson,Gangson",
      "email": "clangsdon0@hc360.com"
    }
]

Index header

If the header is not on the first line you can define the header index like:

 csvToJson.indexHeader(3)
            .getJsonFromCsv(fileInputName);

Empty rows

Empty rows are ignored and not parsed.

Format property value by type

The formatValueByType() function intelligently converts string values to their appropriate types while preserving data integrity. To enable automatic type conversion:

csvToJson.formatValueByType()
         .getJsonFromCsv(fileInputName);

This conversion follows these rules:

Numbers

• Regular integers and decimals are converted to Number type
• Numbers with leading zeros are preserved as strings (e.g., “0012” stays “0012”)
• Large integers outside JavaScript’s safe range are preserved as strings
• Valid decimal numbers are converted to Number type

For example:

{
  "normalInteger": 42,           // Converted to number
  "decimal": 3.14,              // Converted to number
  "leadingZeros": "0012345",    // Kept as string to preserve leading zeros
  "largeNumber": "9007199254740992"  // Kept as string to preserve precision
}

Boolean

Case-insensitive “true” or “false” strings are converted to boolean values:

{
  "registered": true,     // From "true" or "TRUE" or "True"
  "active": false        // From "false" or "FALSE" or "False"
}

Complete Example

Input CSV:

first_name;last_name;email;gender;age;id;zip;registered
Constantin;Langsdon;clangsdon0@hc360.com;Male;96;00123;123;true
Norah;Raison;nraison1@wired.com;Female;32;987;00456;FALSE

Output JSON:

[
 {
  "first_name": "Constantin",
  "last_name": "Langsdon",
  "email": "clangsdon0@hc360.com",
  "gender": "Male",
  "age": 96,
  "id": "00123",        // Preserved leading zeros
  "zip": 123,          // Converted to number
  "registered": true   // Converted to boolean
 },
 {
  "first_name": "Norah",
  "last_name": "Raison",
  "email": "nraison1@wired.com",
  "gender": "Female",
  "age": 32,
  "id": "987",
  "zip": "00456",      // Preserved leading zeros
  "registered": false  // Case-insensitive boolean conversion
 }
]

Encoding

You can read and decode files with the following encoding:

• utf8:

    csvToJson.utf8Encoding()
               .getJsonFromCsv(fileInputName);
  

• ucs2:

     csvToJson.ucs2Encoding()
               .getJsonFromCsv(fileInputName);
  

• utf16le:

      csvToJson.utf16leEncoding()
                 .getJsonFromCsv(fileInputName);
  

• latin1:

      csvToJson.latin1Encoding()
                 .getJsonFromCsv(fileInputName);
  

• ascii:

      csvToJson.asciiEncoding()
                 .getJsonFromCsv(fileInputName);
  

• base64:

      csvToJson.base64Encoding()
                 .getJsonFromCsv(fileInputName);
  

• hex:

      csvToJson.hexEncoding()
                 .getJsonFromCsv(fileInputName);
  

Working with CSV strings directly

If you have CSV content as a string (for example, from an API response or test data), you can parse it directly without writing to a file:

// Parse CSV string to array of objects
let csvString = 'firstName;lastName\nJohn;Doe\nJane;Smith';
let jsonArray = csvToJson.csvStringToJson(csvString);
// Output: [{"firstName":"John","lastName":"Doe"},{"firstName":"Jane","lastName":"Smith"}]

// Parse CSV string to JSON string (validated)
let jsonString = csvToJson.csvStringToJsonStringified(csvString);
// Output: "[\n {\n  \"firstName\": \"John\",\n  \"lastName\": \"Doe\"\n },\n {\n  \"firstName\": \"Jane\",\n  \"lastName\": \"Smith\"\n }\n]"

Both methods support all configuration options through the chaining pattern:

let jsonArray = csvToJson
  .fieldDelimiter(',')
  .formatValueByType()
  .csvStringToJson(csvString);

Async API Usage

This library provides a Promise-based async API that’s perfect for modern Node.js applications. For a detailed migration guide from sync to async API, see MIGRATION.md.

Basic Async Operations

1. Convert CSV file to JSON: ```js const csvToJson = require(‘convert-csv-to-json’);

// Using Promises csvToJson.getJsonFromCsvAsync(‘input.csv’) .then(json => console.log(json)) .catch(err => console.error(‘Error:’, err));

// Using async/await async function convertCsv() { try { const json = await csvToJson.getJsonFromCsvAsync(‘input.csv’); console.log(json); } catch (err) { console.error(‘Error:’, err); } }

2. Generate JSON file from CSV:
```js
// Using async/await with chain configuration
async function convertAndSave() {
  await csvToJson
    .fieldDelimiter(',')
    .formatValueByType()
    .generateJsonFileFromCsvAsync('input.csv', 'output.json');
}

Working with Raw CSV Data

Process CSV data from memory or network sources:

// Example: Processing CSV from an API
async function processCsvFromApi() {
  const response = await fetch('https://api.example.com/data.csv');
  const csvText = await response.text();
  
  const json = await csvToJson
    .formatValueByType()
    .getJsonFromCsvAsync(csvText, { raw: true });
    
  return json;
}

Processing Large Files

For large files, use streaming to manage memory efficiently:

const { createReadStream } = require('fs');
const { createInterface } = require('readline');

async function* processLargeFile(filePath) {
  const fileStream = createReadStream(filePath);
  const rl = createInterface({
    input: fileStream,
    crlfDelay: Infinity
  });

  for await (const line of rl) {
    yield await csvToJson.getJsonFromCsvAsync(line, { raw: true });
  }
}

// Usage
async function processData() {
  for await (const record of processLargeFile('large.csv')) {
    await saveToDatabase(record);
  }
}

Error Handling and Retries

Implement robust error handling with retries:

async function processWithRetry(filePath, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      const json = await csvToJson
        .formatValueByType()
        .getJsonFromCsvAsync(filePath);
      
      return json;
    } catch (err) {
      if (i === maxRetries - 1) throw err;
      // Exponential backoff
      await new Promise(resolve => 
        setTimeout(resolve, Math.pow(2, i) * 1000)
      );
    }
  }
}

Batch Processing

Process multiple files efficiently:

async function batchProcess(files, batchSize = 3) {
  const results = new Map();
  
  for (let i = 0; i < files.length; i += batchSize) {
    const batch = files.slice(i, i + batchSize);
    const processed = await Promise.all(
      batch.map(async file => {
        const json = await csvToJson.getJsonFromCsvAsync(file);
        return [file, json];
      })
    );
    
    processed.forEach(([file, json]) => results.set(file, json));
  }
  
  return results;
}

// Usage
const files = ['data1.csv', 'data2.csv', 'data3.csv', 'data4.csv'];
const results = await batchProcess(files, 2);

Chaining Pattern

The exposed API is implemented with the Method Chaining Pattern, which means that multiple methods can be chained. This pattern works with both synchronous and asynchronous methods:

Synchronous Chaining

const csvToJson = require('convert-csv-to-json');

// Chain configuration methods with sync operation
const json = csvToJson
    .fieldDelimiter(',')
    .formatValueByType()
    .parseSubArray("*", ',')
    .supportQuotedField(true)
    .getJsonFromCsv('myInputFile.csv');

// Chain with file generation
csvToJson
    .fieldDelimiter(';')
    .utf8Encoding()
    .formatValueByType()
    .generateJsonFileFromCsv('input.csv', 'output.json');

// Chain with string parsing
const jsonArray = csvToJson
    .fieldDelimiter(',')
    .trimHeaderFieldWhiteSpace(true)
    .csvStringToJson('name,age\nJohn,30\nJane,25');

Asynchronous Chaining

const csvToJson = require('convert-csv-to-json');

// Using async/await
async function processCSV() {
    // Chain configuration methods with async operation
    const json = await csvToJson
        .fieldDelimiter(',')
        .formatValueByType()
        .parseSubArray("*", ',')
        .supportQuotedField(true)
        .getJsonFromCsvAsync('myInputFile.csv');

    // Chain with async file generation
    await csvToJson
        .fieldDelimiter(';')
        .utf8Encoding()
        .formatValueByType()
        .generateJsonFileFromCsvAsync('input.csv', 'output.json');
}

// Using Promises
csvToJson
    .fieldDelimiter(',')
    .formatValueByType()
    .getJsonFromCsvAsync('input.csv')
    .then(json => console.log(json))
    .catch(err => console.error('Error:', err));

All configuration methods can be chained in any order before calling the final operation method (like getJsonFromCsv, getJsonFromCsvAsync, etc.). The configuration will be applied in the order it is chained.

Common Use Cases

Here are some common use cases and how to implement them:

1. Processing CSV from HTTP Response

const https = require('https');

async function processRemoteCsv(url) {
  const csvData = await new Promise((resolve, reject) => {
    https.get(url, (res) => {
      let data = '';
      res.on('data', chunk => data += chunk);
      res.on('end', () => resolve(data));
      res.on('error', reject);
    });
  });

  return csvToJson.getJsonFromCsvAsync(csvData, { raw: true });
}

2. Batch Processing Multiple Files

async function batchProcess(files) {
  const results = new Map();
  
  // Process in chunks of 3 files at a time
  for (let i = 0; i < files.length; i += 3) {
    const chunk = files.slice(i, i + 3);
    const processed = await Promise.all(
      chunk.map(async file => {
        const json = await csvToJson.getJsonFromCsvAsync(file);
        return [file, json];
      })
    );
    
    processed.forEach(([file, json]) => results.set(file, json));
  }
  
  return results;
}

3. Data Transformation Pipeline

async function transformData(csvFile) {
  // Step 1: Parse CSV
  const json = await csvToJson
    .formatValueByType()
    .getJsonFromCsvAsync(csvFile);
    
  // Step 2: Transform data
  const transformed = json.map(record => ({
    id: record.id,
    fullName: `${record.firstName} ${record.lastName}`,
    age: Number(record.age),
    isAdult: Number(record.age) >= 18,
    email: record.email.toLowerCase()
  }));
  
  // Step 3: Filter invalid records
  return transformed.filter(record => 
    record.id && 
    record.fullName.length > 0 &&
    !isNaN(record.age)
  );
}

4. Error Recovery and Logging

async function processWithLogging(file) {
  const logger = {
    info: (msg) => console.log(`[INFO] ${msg}`),
    error: (msg, err) => console.error(`[ERROR] ${msg}`, err)
  };

  try {
    logger.info(`Starting processing ${file}`);
    const startTime = Date.now();
    
    const json = await csvToJson.getJsonFromCsvAsync(file);
    
    const duration = Date.now() - startTime;
    logger.info(`Processed ${file} in ${duration}ms`);
    
    return json;
  } catch (err) {
    logger.error(`Failed to process ${file}`, err);
    throw err;
  }
}

Troubleshooting

Here are solutions to common issues you might encounter:

Memory Issues with Large Files

If you’re processing large CSV files and encountering memory issues:

// Instead of loading the entire file
const json = await csvToJson.getJsonFromCsvAsync('large.csv'); // ❌

// Use streaming with async iteration
for await (const record of processLargeCsv('large.csv')) {  // ✅
  // Process one record at a time
  await processRecord(record);
}

Handling Different CSV Formats

1. Mixed Quote Types:

   csvToJson
     .supportQuotedField(true)     // Enable quoted field support
     .getJsonFromCsvAsync(file);
   

2. Custom Delimiters:

   csvToJson
     .fieldDelimiter(';')          // Change delimiter
     .getJsonFromCsvAsync(file);
   

3. UTF-8 with BOM:

   csvToJson
     .encoding('utf8')             // Specify encoding
     .getJsonFromCsvAsync(file);
   

Common Error Solutions

1. ENOENT: no such file or directory
   • Check if the file path is correct and absolute
   • Verify file permissions
   • Ensure the file exists in the specified location
2. Invalid CSV Structure
   • Verify CSV format matches expected structure
   • Check for missing or extra delimiters
   • Validate header row exists if expected
3. Memory Leaks
   • Use streaming for large files
   • Process files in smaller chunks
   • Implement proper cleanup in try/finally blocks
4. Encoding Issues
   • Specify correct encoding using .encoding()
   • Check for BOM markers
   • Verify source file encoding

Performance Optimization

1. Parallel Processing: ```js // Instead of sequential processing for (const file of files) { await process(file); // ❌ }

// Use parallel processing with limits async function processWithLimit(files, limit = 3) { const results = []; for (let i = 0; i < files.length; i += limit) { const chunk = files.slice(i, i + limit); const chunkResults = await Promise.all( chunk.map(file => csvToJson.getJsonFromCsvAsync(file)) ); results.push(…chunkResults); } return results; } // ✅

2. **Memory Usage**:
```js
// Clear references when done
async function processWithCleanup(file) {
  let json;
  try {
    json = await csvToJson.getJsonFromCsvAsync(file);
    return await processData(json);
  } finally {
    json = null;  // Clear reference
  }
}

TypeScript Support

If you’re using TypeScript and encounter type issues:

// Define custom types for your CSV structure
interface MyCsvRecord {
  id: number;
  name: string;
  age?: number;
}

// Use type assertion
const json = await csvToJson.getJsonFromCsvAsync<MyCsvRecord>('data.csv');

Development

• Download all csvToJson dependencies:

    npm install
  

• Run Tests

    npm test
  

• Debug Tests

    npm run test-debug
  

CI CD github action

This repository uses the GitHub Action iuccio/npm-semantic-publish-action@latest to publish the npm packeges. Pushing on the master branch, depending on the git message, an new version will always be released. If the commit message contains the keyword:

• [MAJOR]: new major relase, e.g. v1.0.0 -> v2.0.0
• [PATCH]: new patch relase, e.g. v1.0.0 -> v1.0.1
• without any of the above keywords a new minor relase will be applied, e.g. v1.0.0 -> v1.1.0

License

CSVtoJSON is licensed under the MIT License.

Buy me a Coffee

Just if you want to support this repository:

• BTC tip address: 37vdjQhbaR7k7XzhMKWzMcnqUxfw1njBNk