"""
Output of my_data_hasher.py for shingle size = 5

/usr/bin/python /Users/priyanka/Desktop/Data_Mining/Code/mining-project/my_data_hasher.py
Shingling articles...

Shingling 19043 docs took 4.09 sec.

Average shingles per doc: 135.83

Calculating Jaccard Similarities...
  (0 / 19043)
  (1000 / 19043)
  (2000 / 19043)
  (3000 / 19043)
  (4000 / 19043)
  (5000 / 19043)
  (6000 / 19043)
  (7000 / 19043)
  (8000 / 19043)
  (9000 / 19043)
  (10000 / 19043)
  (11000 / 19043)
  (12000 / 19043)
  (13000 / 19043)
  (14000 / 19043)
  (15000 / 19043)
  (16000 / 19043)
  (17000 / 19043)
  (18000 / 19043)
  (19000 / 19043)

Calculating all Jaccard Similarities took 7895.25sec

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 134.13sec

Comparing all signatures...

Comparing MinHash signatures took 15896.51sec
For k= 16

Sum is : 11616.0547592

SSE is: 0.609990797626

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 87.76sec

Comparing all signatures...

Comparing MinHash signatures took 1389.37sec
For k= 32

Sum is : 4564.08131359

SSE is: 0.239672389518

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 100.92sec

Comparing all signatures...

Comparing MinHash signatures took 1511.58sec
For k= 64

Sum is : 1200.59608341

SSE is: 0.0630465831753

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 905.01sec

Comparing all signatures...

Comparing MinHash signatures took 8218.67sec
For k= 128

Sum is : 655.386902054

SSE is: 0.0344161582762

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 148.52sec

Comparing all signatures...

Comparing MinHash signatures took 11661.31sec
For k= 256

Sum is : 351.326530347

SSE is: 0.018449116754

Output of my_data_hasher.py for shingle size = 3

/usr/bin/python /Users/priyanka/Desktop/Data_Mining/Code/mining-project/my_data_hasher.py
Shingling articles...

Shingling 19043 docs took 3.31 sec.

Average shingles per doc: 137.83

Calculating Jaccard Similarities...
  (0 / 19043)
  (1000 / 19043)
  (2000 / 19043)
  (3000 / 19043)
  (4000 / 19043)
  (5000 / 19043)
  (6000 / 19043)
  (7000 / 19043)
  (8000 / 19043)
  (9000 / 19043)
  (10000 / 19043)
  (11000 / 19043)
  (12000 / 19043)
  (13000 / 19043)
  (14000 / 19043)
  (15000 / 19043)
  (16000 / 19043)
  (17000 / 19043)
  (18000 / 19043)
  (19000 / 19043)

Calculating all Jaccard Similarities took 15022.51sec

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 63.38sec

Comparing all signatures...

Comparing MinHash signatures took 737.09sec
For k= 16

Sum is : 87813.9705493

SSE is: 4.61135170662

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 99.72sec

Comparing all signatures...

Comparing MinHash signatures took 12839.62sec
For k= 32

Sum is : 67206.5986154

SSE is: 3.52920225885

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 115.70sec

Comparing all signatures...

Comparing MinHash signatures took 4433.85sec
For k= 64

Sum is : 14181.1968037

SSE is: 0.744693420349

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 67.07sec

Comparing all signatures...

Comparing MinHash signatures took 2751.41sec
For k= 128

Sum is : 8127.06311283

SSE is: 0.426774306193

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 153.00sec

Comparing all signatures...


Comparing MinHash signatures took 6269.49sec
For k= 256

Sum is : 4624.31923291

SSE is: 0.242835647372

Output of my_data_hasher.py for shingle size = 2

Shingling articles...

Shingling 19043 docs took 2.93 sec.

Average shingles per doc: 138.83

Calculating Jaccard Similarities...
  (0 / 19043)
  (1000 / 19043)
  (2000 / 19043)
  (3000 / 19043)
  (4000 / 19043)
  (5000 / 19043)
  (6000 / 19043)
  (7000 / 19043)
  (8000 / 19043)
  (9000 / 19043)
  (10000 / 19043)
  (11000 / 19043)
  (12000 / 19043)
  (13000 / 19043)
  (14000 / 19043)
  (15000 / 19043)
  (16000 / 19043)
  (17000 / 19043)
  (18000 / 19043)
  (19000 / 19043)

Calculating all Jaccard Similarities took 2403.89sec

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 57.11sec

Comparing all signatures...

Comparing MinHash signatures took 558.17sec
For k= 16

Sum is : 109748.379516

SSE is: 5.76318749756

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 16.33sec

Comparing all signatures...

Comparing MinHash signatures took 5210.33sec
For k= 32

Sum is : 69422.8847926

SSE is: 3.6455855061

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 115.06sec

Comparing all signatures...

Comparing MinHash signatures took 1650.50sec
For k= 64

Sum is : 42278.3568361

SSE is: 2.22015212079

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 67.42sec

Comparing all signatures...

Comparing MinHash signatures took 4679.09sec
For k= 128

Sum is : 16905.9646357

SSE is: 0.887778429644

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 316.01sec

Comparing all signatures...

Comparing MinHash signatures took 6083.50sec
For k= 256

Sum is : 7029.37510354

SSE is: 0.369131707375

Output of my_data_hasher.py for shingle size = 1

Shingling articles...

Shingling 19043 docs took 2.69 sec.

Average shingles per doc: 139.83

Calculating Jaccard Similarities...
  (0 / 19043)
  (1000 / 19043)
  (2000 / 19043)
  (3000 / 19043)
  (4000 / 19043)
  (5000 / 19043)
  (6000 / 19043)
  (7000 / 19043)
  (8000 / 19043)
  (9000 / 19043)
  (10000 / 19043)
  (11000 / 19043)
  (12000 / 19043)
  (13000 / 19043)
  (14000 / 19043)
  (15000 / 19043)
  (16000 / 19043)
  (17000 / 19043)
  (18000 / 19043)
  (19000 / 19043)

Calculating all Jaccard Similarities took 3233.74sec

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 71.76sec

Comparing all signatures...

Comparing MinHash signatures took 644.76sec
For k= 16

Sum is : 489545.087547

SSE is: 25.7073511289

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 13.59sec

Comparing all signatures...

Comparing MinHash signatures took 912.10sec
For k= 32

Sum is : 282334.342258

SSE is: 14.8261483095

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 99.07sec

Comparing all signatures...

Comparing MinHash signatures took 1439.87sec
For k= 64

Sum is : 144349.348711

SSE is: 7.58017900074

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 45.92sec

Comparing all signatures...

Comparing MinHash signatures took 2786.62sec
For k= 128

Sum is : 73703.6818729

SSE is: 3.87038186593

Generating random hash functions...

Generating MinHash signatures for all documents...

Generating MinHash signatures took 107.07sec

Comparing all signatures...

Comparing MinHash signatures took 5310.45sec
For k= 256

Sum is : 55522.9066507

SSE is: 2.91565964663

"""

import matplotlib.pyplot as plt

hash_range = [16,32,64,128,256]
sse = [4.61135170662,3.52920225885, 0.744693420349,  0.426774306193 , 0.242835647372]

efficacy = [737.09 , 12839.62 , 4433.85 , 2751.41 , 6269.49 ]

plt.plot(hash_range, sse, 'xb-')
plt.axis([5, 280, min(sse) - 0.2, max(sse) + 1])
plt.show()

plt.plot(hash_range, efficacy, 'xr-')
plt.axis([5, 280, min(efficacy) - 100, max(efficacy) + 50])
plt.show()
