Assignment 1 - Summarization

Report/Code Due Date: 2020/04/07 23:59

Reach Baseline

  • Implement extractive summarization and abstractive summarization w/ and w/o attention
  • Beat the three baselines

Early Bird Bonus

  • Submit either extractive or abstractive model before early bird deadline
  • TAs will release code for preprocessing after early bird deadline

Deadline

  • Early Bird Bonus: 2020/03/25 23:59
  • Report/Code Submission: 2020/04/07 23:59

Assignment 1 (20% + BONUS 2%)

If there is any conflict or unclear things in the following, please follow the rules defined in the slides.

Small tips: you can use README.md to record all the progress you done, at the time you just do it. Rearrange it after you finish your assignment.

Reach Baseline (9%)

You need to implement sequence tagging and seq2seq models which take full articles as input and select sentences in the article as summary or generate a summary, respectively.
Dataset description
Please see slides for more details
The input format:

{
	"id": "1000000",
	"summary": "A seven-hundred-year old oak gate at Salisbury Cathedral ...",
	"text": "The Grade I listed Harnham Gate was hit by a white van that smashed into the structure at about 02:00 BST ...",
	"sent_bounds": [[0, 107], [107, 255], [255, 362]],
	"extractive_summary": 1
}
Test set will not have the "summary" and "extractive_summary" entry.

We adopt jsonl (lines of json) as our output format.
Extractive outputs should contain columns "id" and "predict_sentence_index", and abstractive outputs contain "id" and "predict". The extractive output format:

{"id": "2000000", "predict_sentence_index": [0, 1, 3, 10]}
{"id": "2000001", "predict_sentence_index": [3]}
...
The abstractive output format:

{"id": "2000000", "predict": "Your summary of the articles"}
{"id": "2000001", "predict": "should be here."}
...
Evaluation:
TA will test your model as follow:
bash ./early.sh /path/to/test.jsonl /path/to/predict.jsonl
You need to specify the Python version (only 3.6, 3.7) in the .sh file.
Ex. python3.7 predict.py
Please check NTU COOL for the rouge score evaluation scripts.

Report (10% + BONUS 2%)

  • Q0: Format (1%)
  • Your will get the point if
    1. Your report file is pdf.
    2. The path to your file is correct.
  • Q1: Describe how do you use the data. (1%)
    1. How do you tokenize the data?
    2. Truncation length of the text and the summary.
    3. The pre-trained embedding you used.
  • Q2: Describe your Extractive Summarization model. (2%)
    • Describe
      • your model.
      • performance of your model. (on the val. set)
      • the loss function you used.
      • the optimization algorithm, learning rate and batch size.
      • post-processing strategy.
  • Q3: Describe your Seq2Seq + Attention model. (2%)
    • Describe
      • your model.
      • performance of your model. (on the val. set)
      • the loss function you used.
      • the optimization algorithm, learning rate and batch size.
  • Q4: Plot the distribution of relative locations of your predicted sentences by your extractive model. (1%)
      Please records how to reproduce your figure in README.
  • Q5: Visualize the attention weights. (2%)
    1. Take one example in the validation set and visualize the attention weights (after softmax).
      • Readable text on the two axises. (0.5%)
      • Colors that indicate the value. (0.5%)
    2. Describe your findings. (1%)
    Reminder: You can use any publicly available tool/package/code to plot the figure, but don't import the package in your predicting script.
  • Q6: Explain Rouge-L. (1%)
    1. Explain thw way Rouge-L is calculated.
    You don't need to explain what is covered in the ADA(Algorithm Design and Analysis) course.
  • Q7: Beam Search. (BONUS 2%)
    1. Implement it, and write pseudocode of beam search on report
    2. For both seq2se2 w/ and w/o attention, compare the performance of
      • decoding without beam search.
      • decoding with beam search.
    3. Give samples where beam search performs significantly better.
    4. Give reasons why beam search may perform better.

Format (1%)

TA can run the grading script without human intervention.

Early Bird Bonus(1%)

Before 3/25 23:59, submit a model either of extractive, seq2seq, seq2seq + attention model that can generate some prediction.

TA Team

Please contact TAs for assistance.
Links for online TA hours would be shared at NTU COOL.

user

江廷睿

General questions

TA Hour: Tue 11:00-12:00, Wed 11:00-12:00
Location: Tue@德田102, Wed@Online
user

林彥廷

General questions

TA Hour: Tue 11:00-12:00, Wed 11:00-12:00
Location: Tue@德田102, Wed@Online

What's New?

Mar 3 - Website launch.

Mar 17 - Assignment 1 released.