CSS Tokenizer

Implemented from : https://www.w3.org/TR/2021/CRD-css-syntax-3-20211224/

API

Read the API docs

Usage

Add CSS Tokenizer to your project:

 npm install @csstools/css-tokenizer --save-dev

 import { tokenizer, TokenType } from '@csstools/css-tokenizer';

const myCSS = `@media only screen and (min-width: 768rem) {
	.foo {
		content: 'Some content!' !important;
	}
}
`;

const t = tokenizer({
	css: myCSS,
});

while (true) {
	const token = t.nextToken();
	if (token[0] === TokenType.EOF) {
		break;
	}

	console.log(token);
}

Or use the tokenize helper function:

 import { tokenize } from '@csstools/css-tokenizer';

const myCSS =  `@media only screen and (min-width: 768rem) {
	.foo {
		content: 'Some content!' !important;
	}
}
`;

const tokens = tokenize({
	css: myCSS,
});

console.log(tokens);

Options

 {
	onParseError?: (error: ParseError) => void
}

onParseError

The tokenizer is forgiving and won't stop when a parse error is encountered.

To receive parsing error information you can set a callback.

 import { tokenizer, TokenType } from '@csstools/css-tokenizer';

const t = tokenizer({
	css: '\\',
}, { onParseError: (err) => console.warn(err) });

while (true) {
	const token = t.nextToken();
	if (token[0] === TokenType.EOF) {
		break;
	}
}

Parser errors will try to inform you where in the tokenizer logic the error happened.
This tells you what kind of error occurred.

Goals and non-goals

Things this package aims to be:

• specification compliant CSS tokenizer
• a reliable low level package to be used in CSS parsers

What it is not:

• opinionated
• fast
• small