I should use the shell more often

How I leverage command-line tool to solve the problem I encounter at work.

Background

I was given an assignment today. Our product will launch on Product Hunt tomorrow.

It's a milestone for us and we need to send an email to tell our users we hope they give us support and vote for us on Product Hunt.

Who are we gonna mail?

Our target users will be the monthly active users who used our product in the past month.

I'm not the one with access to this data so I asked and got a .csv file containing the user_id to look up the corresponding data in the database.

How do we send email?

We run a Node.js script with Nodemailer and Amazon Simple Email Service(SES) to send emails, querying the database to exclude unsubscribed users.

for (const ids of userIds) {
  const users = await queryInterface.sequelize.query(
    `SELECT
			users.id,
			users.email,
		FROM
			users
			LEFT JOIN user_data ON users.id = user_data.user_id
		WHERE
			users.id in (${ids})
		AND users.email != ''
		AND users.email IS NOT NULL
		AND user_data.preferences NOT LIKE '%\"emailSubscriptionState\":\"UNSUBSCRIBED\"%'
		ORDER BY users.id ASC LIMIT 10`,
    {
      raw: true,
      type: queryInterface.sequelize.QueryTypes.SELECT,
    },
  );
  await Promise.allSettled(users.map(sendEmailFn));
  await waitOneSecond();
}

The problem

What I need to do is convert the userId in the .csv file to a javascript array and break the array into small chunks, each of them a size of 10.

The .csv file looks like this.

"user_id"
"123"
"321"
"234"
"543"
"8667"

What I need is each userId separate with , so I can just use String.prototype.split() to transform them to an array.

Obviously, I can just use Node.js to process the file since Node.js provides a built-in fs module to let our developer deal with this situation.

But I've been relying so much on Node.js that I've forgotten how much Shell can do.

With the built-in command line tool we can easily achieve our requirements.

The solution

To transform the CSV file content to what we want. We can break into the following steps

• remove "user_id"
• remove "
• replace \n with ,

cat test.csv | sed '1d' | sed 's/"//g' | tr '\n' ',' > test.txt

We pipe the output from the cat command into the sed command and then into the next one using the pipe operator (which is | or the pipe symbol)

Lastly, we will redirect the result into the test.txt file with the > symbol.

The output will look like 123,321,234,543,8667,.

We will get [ "123", "321", "234", "543", "8667", "" ] if we split this string in javascript. An extra empty string is not what we want.

cat test.csv | sed '1d' | sed 's/"//g' | tr '\n' ',' | sed 's/,$//' > test.txt

We remove the last comma in the string and we are good to go.

I should use the shell more often instead of just npm run script or git commit -m "😅".

Reference

Effective Shell

sed-script-overview