This repository contains Jupyter notebooks that demonstrate the process of building and evaluating machine learning models for automated essay scoring.
The goal is to create reliable and efficient models that can provide timely feedback to students and support educators, especially in underserved communities.

This approach involves using LightGBM (Light Gradient Boosting Machine) with TF-IDF (Term Frequency-Inverse Document Frequency) for feature extraction to classify essay scores.

LightGBM is a gradient boosting framework that uses tree-based learning algorithms. It is designed to be distributed and efficient, with the capability to handle large-scale data with high performance and speed. In this project, LightGBM is used to build a model that can accurately predict essay scores based on features extracted from the text data.

TF-IDF is a statistical measure used to evaluate the importance of a word in a document relative to a collection of documents. It is a numerical representation of text that transforms the raw text into features that can be used by machine learning algorithms. By applying TF-IDF, the textual data from essays is converted into a format that LightGBM can process, capturing the relevance and significance of each term within the essays.

Combining LightGBM with TF-IDF allows for an effective classification model that can process and analyze the text data to predict scores accurately. This method leverages the strengths of LightGBM in handling large datasets and the ability of TF-IDF to highlight important textual features, resulting in a robust automated essay scoring system.

RAPIDS SVR (Support Vector Regression) is utilized for regression tasks on large datasets, leveraging GPU acceleration for faster computation. RAPIDS is an open-source suite of software libraries and APIs built on CUDA, which enables execution on NVIDIA GPUs.

SVR is a type of Support Vector Machine (SVM) used for regression challenges. It seeks to find a function that deviates from the actual observed values by a value no greater than a specified margin and at the same time is as flat as possible. SVR is particularly effective in cases where the relationship between data points is complex and non-linear.

Using RAPIDS SVR in this project allows for efficient handling and processing of large-scale essay datasets, enabling faster training and prediction times due to the parallel computing capabilities of GPUs. This makes it feasible to work with extensive data, ensuring that the regression model can provide precise scoring predictions in a timely manner.

This project was bootstrapped with Create React App.

In the project directory, you can run:

Runs the app in the development mode.
Open http://localhost:3000 to view it in your browser.

The page will reload when you make changes.
You may also see any lint errors in the console.

Launches the test runner in the interactive watch mode.
See the section about running tests for more information.

Builds the app for production to the build folder.
It correctly bundles React in production mode and optimizes the build for the best performance.

The build is minified and the filenames include the hashes.
Your app is ready to be deployed!

See the section about deployment for more information.

Note: this is a one-way operation. Once you eject, you can’t go back!

If you aren’t satisfied with the build tool and configuration choices, you can eject at any time. This command will remove the single build dependency from your project.

Instead, it will copy all the configuration files and the transitive dependencies (webpack, Babel, ESLint, etc) right into your project so you have full control over them. All of the commands except eject will still work, but they will point to the copied scripts so you can tweak them. At this point you’re on your own.

You don’t have to ever use eject. The curated feature set is suitable for small and middle deployments, and you shouldn’t feel obligated to use this feature. However we understand that this tool wouldn’t be useful if you couldn’t customize it when you are ready for it.

You can learn more in the Create React App documentation.

To learn React, check out the React documentation.

This section has moved here: https://facebook.github.io/create-react-app/docs/code-splitting

This section has moved here: https://facebook.github.io/create-react-app/docs/analyzing-the-bundle-size

This section has moved here: https://facebook.github.io/create-react-app/docs/making-a-progressive-web-app

This section has moved here: https://facebook.github.io/create-react-app/docs/advanced-configuration

This section has moved here: https://facebook.github.io/create-react-app/docs/deployment

This section has moved here: https://facebook.github.io/create-react-app/docs/troubleshooting#npm-run-build-fails-to-minify

mdast extensions to parse and serialize frontmatter (YAML, TOML, and more).

• What is this?
• When to use this
• Install
• Use
• API
  • frontmatterFromMarkdown(options?)
  • frontmatterToMarkdown(options?)
  • Info
  • Matter
  • Options
• Syntax
• Syntax tree
  • Nodes
  • Content model
• Types
• Compatibility
• Related
• Contribute
• License

This package contains two extensions that add support for frontmatter syntax as often used in markdown to mdast. These extensions plug into mdast-util-from-markdown (to support parsing frontmatter in markdown into a syntax tree) and mdast-util-to-markdown (to support serializing frontmatter in syntax trees to markdown).

Frontmatter is a metadata format in front of the content. It’s typically written in YAML and is often used with markdown. Frontmatter does not work everywhere so it makes markdown less portable.

These extensions follow how GitHub handles frontmatter. GitHub only supports YAML frontmatter, but these extensions also support different flavors (such as TOML).

You can use these extensions when you are working with mdast-util-from-markdown and mdast-util-to-markdown already.

When working with mdast-util-from-markdown, you must combine this package with micromark-extension-frontmatter.

When you don’t need a syntax tree, you can use micromark directly with micromark-extension-frontmatter.

All these packages are used remark-frontmatter, which focusses on making it easier to transform content by abstracting these internals away.

This package is ESM only. In Node.js (version 16+), install with npm:

npm install mdast-util-frontmatter

In Deno with esm.sh:

import {frontmatterFromMarkdown, frontmatterToMarkdown} from 'https://esm.sh/mdast-util-frontmatter@2'

In browsers with esm.sh:

<script type="module">
  import {frontmatterFromMarkdown, frontmatterToMarkdown} from 'https://esm.sh/mdast-util-frontmatter@2?bundle'
</script>

Say our document example.md contains:

+++
title = "New Website"
+++

# Other markdown

…and our module example.js looks as follows:

import fs from 'node:fs/promises'
import {frontmatter} from 'micromark-extension-frontmatter'
import {fromMarkdown} from 'mdast-util-from-markdown'
import {frontmatterFromMarkdown, frontmatterToMarkdown} from 'mdast-util-frontmatter'
import {toMarkdown} from 'mdast-util-to-markdown'

const doc = await fs.readFile('example.md')

const tree = fromMarkdown(doc, {
  extensions: [frontmatter(['yaml', 'toml'])],
  mdastExtensions: [frontmatterFromMarkdown(['yaml', 'toml'])]
})

console.log(tree)

const out = toMarkdown(tree, {extensions: [frontmatterToMarkdown(['yaml', 'toml'])]})

console.log(out)

…now running node example.js yields (positional info removed for brevity):

{
  type: 'root',
  children: [
    {type: 'toml', value: 'title = "New Website"'},
    {
      type: 'heading',
      depth: 1,
      children: [{type: 'text', value: 'Other markdown'}]
    }
  ]
}

+++
title = "New Website"
+++

# Other markdown

This package exports the identifiers frontmatterFromMarkdown and frontmatterToMarkdown. There is no default export.

Create an extension for mdast-util-from-markdown.

• options (Options, optional) — configuration

Extension for mdast-util-from-markdown (FromMarkdownExtension).

Create an extension for mdast-util-to-markdown.

• options (Options, optional) — configuration

Extension for mdast-util-to-markdown (ToMarkdownExtension).

Structure of marker or fence (TypeScript type).

Same as Info from micromark-extension-frontmatter.

Structure of matter (TypeScript type).

Same as Matter from micromark-extension-frontmatter.

Configuration (TypeScript type).

Same as Options from micromark-extension-frontmatter.

See Syntax in micromark-extension-frontmatter.

The following interfaces are added to mdast by this utility.

👉 Note: other nodes are not enabled by default, but when passing options to enable them, they work the same as YAML.

interface YAML <: Literal {
  type: "yaml"
}

YAML (Literal) represents a collection of metadata for the document in the YAML data serialization language.

YAML can be used where frontmatter content is expected. Its content is represented by its value field.

For example, the following markdown:

---
foo: bar
---

Yields:

{type: 'yaml', value: 'foo: bar'}

type FrontmatterContent = YAML

Frontmatter content represent out-of-band information about the document.

If frontmatter is present, it must be limited to one node in the tree, and can only exist as a head.

type FlowContentFrontmatter = FrontmatterContent | FlowContent

This package is fully typed with TypeScript. It exports the additional types Info, Matter, and Options.

The YAML node type is supported in @types/mdast by default. To add other node types, register them by adding them to FrontmatterContentMap:

import type {Literal} from 'mdast'

interface Toml extends Literal {
  type: 'toml'
}

declare module 'mdast' {
  interface FrontmatterContentMap {
    // Allow using TOML nodes defined by `mdast-util-frontmatter`.
    toml: Toml
  }
}

Projects maintained by the unified collective are compatible with maintained versions of Node.js.

When we cut a new major release, we drop support for unmaintained versions of Node. This means we try to keep the current release line, mdast-util-frontmatter@^2, compatible with Node.js 16.

This utility works with mdast-util-from-markdown version 2+ and mdast-util-to-markdown version 2+.

• remark-frontmatter — remark plugin to support frontmatter
• micromark-extension-frontmatter — micromark extension to parse frontmatter

See contributing.md in syntax-tree/.github for ways to get started. See support.md for ways to get help.

This project has a code of conduct. By interacting with this repository, organization, or community you agree to abide by its terms.

MIT © Titus Wormer

mdast extensions to parse and serialize frontmatter (YAML, TOML, and more).

• What is this?
• When to use this
• Install
• Use
• API
  • frontmatterFromMarkdown(options?)
  • frontmatterToMarkdown(options?)
  • Info
  • Matter
  • Options
• Syntax
• Syntax tree
  • Nodes
  • Content model
• Types
• Compatibility
• Related
• Contribute
• License

This package contains two extensions that add support for frontmatter syntax as often used in markdown to mdast. These extensions plug into mdast-util-from-markdown (to support parsing frontmatter in markdown into a syntax tree) and mdast-util-to-markdown (to support serializing frontmatter in syntax trees to markdown).

Frontmatter is a metadata format in front of the content. It’s typically written in YAML and is often used with markdown. Frontmatter does not work everywhere so it makes markdown less portable.

These extensions follow how GitHub handles frontmatter. GitHub only supports YAML frontmatter, but these extensions also support different flavors (such as TOML).

You can use these extensions when you are working with mdast-util-from-markdown and mdast-util-to-markdown already.

When working with mdast-util-from-markdown, you must combine this package with micromark-extension-frontmatter.

When you don’t need a syntax tree, you can use micromark directly with micromark-extension-frontmatter.

All these packages are used remark-frontmatter, which focusses on making it easier to transform content by abstracting these internals away.

This package is ESM only. In Node.js (version 16+), install with npm:

npm install mdast-util-frontmatter

In Deno with esm.sh:

import {frontmatterFromMarkdown, frontmatterToMarkdown} from 'https://esm.sh/mdast-util-frontmatter@2'

In browsers with esm.sh:

<script type="module">
  import {frontmatterFromMarkdown, frontmatterToMarkdown} from 'https://esm.sh/mdast-util-frontmatter@2?bundle'
</script>

Say our document example.md contains:

+++
title = "New Website"
+++

# Other markdown

…and our module example.js looks as follows:

import fs from 'node:fs/promises'
import {frontmatter} from 'micromark-extension-frontmatter'
import {fromMarkdown} from 'mdast-util-from-markdown'
import {frontmatterFromMarkdown, frontmatterToMarkdown} from 'mdast-util-frontmatter'
import {toMarkdown} from 'mdast-util-to-markdown'

const doc = await fs.readFile('example.md')

const tree = fromMarkdown(doc, {
  extensions: [frontmatter(['yaml', 'toml'])],
  mdastExtensions: [frontmatterFromMarkdown(['yaml', 'toml'])]
})

console.log(tree)

const out = toMarkdown(tree, {extensions: [frontmatterToMarkdown(['yaml', 'toml'])]})

console.log(out)

…now running node example.js yields (positional info removed for brevity):

{
  type: 'root',
  children: [
    {type: 'toml', value: 'title = "New Website"'},
    {
      type: 'heading',
      depth: 1,
      children: [{type: 'text', value: 'Other markdown'}]
    }
  ]
}

+++
title = "New Website"
+++

# Other markdown

This package exports the identifiers frontmatterFromMarkdown and frontmatterToMarkdown. There is no default export.

Create an extension for mdast-util-from-markdown.

• options (Options, optional) — configuration

Extension for mdast-util-from-markdown (FromMarkdownExtension).

Create an extension for mdast-util-to-markdown.

• options (Options, optional) — configuration

Extension for mdast-util-to-markdown (ToMarkdownExtension).

Structure of marker or fence (TypeScript type).

Same as Info from micromark-extension-frontmatter.

Structure of matter (TypeScript type).

Same as Matter from micromark-extension-frontmatter.

Configuration (TypeScript type).

Same as Options from micromark-extension-frontmatter.

See Syntax in micromark-extension-frontmatter.

The following interfaces are added to mdast by this utility.

👉 Note: other nodes are not enabled by default, but when passing options to enable them, they work the same as YAML.

interface YAML <: Literal {
  type: "yaml"
}

YAML (Literal) represents a collection of metadata for the document in the YAML data serialization language.

YAML can be used where frontmatter content is expected. Its content is represented by its value field.

For example, the following markdown:

---
foo: bar
---

Yields:

{type: 'yaml', value: 'foo: bar'}

type FrontmatterContent = YAML

Frontmatter content represent out-of-band information about the document.

If frontmatter is present, it must be limited to one node in the tree, and can only exist as a head.

type FlowContentFrontmatter = FrontmatterContent | FlowContent

This package is fully typed with TypeScript. It exports the additional types Info, Matter, and Options.

The YAML node type is supported in @types/mdast by default. To add other node types, register them by adding them to FrontmatterContentMap:

import type {Literal} from 'mdast'

interface Toml extends Literal {
  type: 'toml'
}

declare module 'mdast' {
  interface FrontmatterContentMap {
    // Allow using TOML nodes defined by `mdast-util-frontmatter`.
    toml: Toml
  }
}

Projects maintained by the unified collective are compatible with maintained versions of Node.js.

When we cut a new major release, we drop support for unmaintained versions of Node. This means we try to keep the current release line, mdast-util-frontmatter@^2, compatible with Node.js 16.

This utility works with mdast-util-from-markdown version 2+ and mdast-util-to-markdown version 2+.

• remark-frontmatter — remark plugin to support frontmatter
• micromark-extension-frontmatter — micromark extension to parse frontmatter

See contributing.md in syntax-tree/.github for ways to get started. See support.md for ways to get help.

This project has a code of conduct. By interacting with this repository, organization, or community you agree to abide by its terms.

MIT © Titus Wormer

I bought 500 Twitter followers from one of the leading providers for this kind of service. What I got can be found in fake_accounts.csv: Users with mostly all having ~1500 Tweets (most of them are retweets) about Trump, Porn, Saudi Arabia. 1/10 would not buy again.