Remove duplicate records from a database in SPMF format (a text file) (SPMF documentation)

This example explains how to remove duplicate records from a database in SPMF format (a text file) using the SPMF open-source data mining library.

How to run this example?

• If you are using the graphical interface, (1) choose the "Remove_duplicate_records" algorithm, (2) choose the input file contextResize.txt (3) and click "Run algorithm".
• If you want to execute this example from the command line, then execute this command:
  java -jar spmf.jar run Remove_duplicate_records contextResize.txt output.txt in a folder containing spmf.jar and the input file contextResize.txt.
• If you are using the source code version of SPMF, launch the file "MainTestRemoveDuplicateRecords.java" in the package ca.pfv.SPMF.tests.

What is this tool?

This tool allows to remove duplicate records (lines) from a dataset file. It keeps only the first occurrence of each unique record and removes all subsequent duplicates.

This is useful for cleaning datasets before applying data mining algorithms. The tool preserves metadata lines (lines that are empty or start with #, %, or @) and copies them to the output file without modification. Only data records are checked for duplicates.

What is the input?

The input is a text file in SPMF format. It could be for example a transaction database, a sequence database, or other types of databases used by algorithms offered in SPMF. Moreover the user has to specify a percentage X.

For example, this is the file contextResize.txt:

@FILETYPE="Text file"
@SOURCE="SPMF SOFTWARE https://philippe-fournier-viger.com/spmf/"
1 3 4
2 3 5
1 2 3 5
2 5
1 2 3 5

This file contains two lines of meta data indicating the file type and source of the data, followed by 5 records (each line is a record) containing list of integers separated by spaces.

What is the output?

The output is a text file with the same format as the input, but with duplicate data records removed. Only the first occurrence of each unique record is kept. In this example, the output is a file output.txt as follows:

@FILETYPE="Text file"
@SOURCE="SPMF SOFTWARE https://philippe-fournier-viger.com/spmf/"
1 3 4
2 3 5
1 2 3 5
2 5

The fifth line of the input file has been removed because it is a duplicate of the third line.