Dreamsongs Wiki

BuyPapersWithMinutes

I mentioned my "vote with minutes" idea to Kathleen Fisher from the ICFP Steering Committee who then asked me to send her a formal proposal to submit to the SC. Here it is, for fairness and completeness. -- MatthiasFelleisen

Proposal: to construct a technical program for ICFP from talks of different lengths

Rationale: Conferences in general and ICFP in particular should become more lively. That is, people should feel free to present interesting ideas that are really nothing more than ideas in addition to experiments and theorems with full-fledged evidence and proofs, respectively. Ideas, even more than experiments and theorems, need feedback, encouragement, pruning. A conference is an ideal place for feedback, encouragement and pruning, because like-minded people are gathered and can quickly see whether an idea is worthwhile or not. -- Also, I believe that allowing more and shorter talks than the usual 20 to 30 half-hour presentations will attract a wider audience and will produce more interaction and more participation. -- Finally, it saves conferences on language design, for example, from rejecting a paper that creates an entire new area with its own conference and workshop system six years later. (No names, but you can all think of your own examples.)

Concrete Implementation Proposal: Here is one concrete implementation proposal. Suggestions for improvement welcome.

(1) We construct three types of sessions: 6 x 10 minutes, 5 x 20 minutes, and 3 x 30 minutes (approximately).
(2) We solicit papers as usual but make it clear in the Call for Papers that a paper may only receive 10 minutes or 20 minutes for a presentation. Authors may also request short slots.
(3) We give each PC members a limited number "dollars" (i.e., minutes) to vote on papers. Any paper that receives at least 10 minutes during the voting is accepted. Period. The longest presentation is limited to 30 minutes, no matter how many votes it receives.

Concrete Example: If you assume 750 minutes of presentation time (3 days x 10 presentationsday x 25 minspresentation) and 12 PC, members, you can give each PC members 75 minutes to vote with. You probably want several rounds of voting during which PC members can reallocate or adjust their votes. Two worst case scenarios may come up:

ONE: All PC members vote all minutes for one paper (unlikely).

TWO: All PC members vote 10 minutes for 7 papers and we get 84 papers into the program. The more the merrier. At 10 mins, we get 840 mins of presentation and around 84 author teams attending.

Experience: I suggested such a system to the Scheme and Functional Programming 2003 workshop, which Matthew Flatt ran and I locally hosted at NU. The attendance went from the usual 35 to 40 to 79 registered attendees. (I paid for the lunches, so I know.) Flatt and his PC picked talks of 15 minutes, 5 mins, and 30 mins.

John Clements and I hosted the NEPLS workshop series here in the spring. We had a selection committee of four (Tim Hickey from Brandeis and someone from Vermont whose name escapes me right now). We constructed sessions like the above and again had the largest NEPLS workshop since its inception. (See nepls.org) People particularly seemed to enjoy the 10 minute session, even if some of the ideas were bogus in there.

I think this is a great idea. A very serious problem with the

way ACM conferences are currently run is that a paper has a

very difficult time making it if it is objectionable to someone

on the program committee. As a result only the blandest, least offensive papers are accepted. The above proposal almost guarantees a much wider range of papers will make it, which is great.

Martin

I think this is worth trying, indeed. Why not try it out with the PractitionerReports 2005? In my opinion, the entry level for practitioner reports is quite high (need to fill a 45 minute talk, project must be finished). If we dedicate two sessions to talks of any length, length determined by the above algorithm, also shorter contributions would be possible; they may be pre-mature, they may be not baked yet, but they might interest some people. At the same time, we also create a venue for PeopleFromIndustry to feel more at home than in the current program.

--tla

Since it's been tried in real conferences before, why not try it for the technical papers in 2005?

So, tell me why I cannot try it? -rpg

I am intrigued by the idea of voting with minutes,

but I have a few issues to raise.

(1) Free flow of information.

A free-market mechanism

such as the stock market assumes that buyers have good

information about the stocks, at least in principle.

The bidding mechanism cannot converge on a proper price

without a reasonably free flow of information.

In the context of paper selection, I think there is a lot of value in

ensuring that PC members have the benefit of each others' expertise in

evaluating the papers, and this flow of information should be

explicitly defined and institutionalized, just as it is for current PC

strategies. (The particular expert information I value is

the identification of a fatal flaw in a superficially appealing

paper, but it is also helpful to have an expert identify the

crucial new idea in a paper that is superficially bland.)

MF says
The PC will still conduct a discussion of the papers.

There will be three groups: those with lots of prelimiary minutes, those

that have fewer than the threshold, and the ones in between. During

those discussion you get the information that you need (well a lot of it)

to revise your "minute votes." Then at the end of this discussion, you vote

again. Now you have a program.

(2) Allocating time at the PC meeting. CyberChair and its "identify

the champion" mechanism allows the recording of both positive and

negative votes (A, B, C, and D, where A and B are positive and C and D

are negative). While we want to have a free flow of information at

the PC meeting, time is a limited resource, and one of the functions

of the PC chair is to guide the allocation of time to the free flow of

information. If "voting with minutes" were to replace "identify the

champion" for the part of the process before the PC meeting, it would

capture positive vote information but not negative vote information,

and therefore fail to identify papers for which further information

flow would be especially helpful. A possible solution is to continue

to use "identify the champion" (or perhaps both methods) before the PC

meeting, but to use "voting with minutes" for the final

decision-making.

MF says
In my limited experience with CyberChair and similar systems,

the negative champion system hardly ever helps much. Also note that

we have had this system of selection for decades, without electronic

support. It got us what we have.

(3) Whiteballing an embarrassing paper. There may be situations

where one committee member is so taken with an off-the-wall paper

that he throws a large number of minutes at it, to the embarrassment

of other program committee members who judge the paper to be severely flawed.

I suggest that there be some limit on the number of minutes that a single

member can throw at a paper, so that an off-the-wall paper can easily

get a short slot, but getting a long slot requires the support of

multiple PC members. If a conference were to have slots of 10, 20 and 30

minutes, perhaps the limit should be 12.

MF says
You're right. I have considered making the names of the

supportive PC members public. My true response is that the value of

conferences as "tenure publication vehicles" will sink and therefore

conferences can tolerate a 30 minute bad paper. People walk out

and the PC knows which referee voted these minutes, even after a

discussion. Let's not invite him again.

(4) Conflicts of interest. Should a PC member be forbidden from

using minutes for any paper for which he has a conflict

of interest in the traditional sense?

MF says
Yes.

(5) PC member papers. Should a paper for which a PC member is a

co-author have to meet a higher standard than usual in order to be

accepted? The obvious "higher standard" would be to have to receive

an extra share of minutes, but then the game is no longer conservative

of minutes; voting in too many PC papers might reduce the pool

of minutes to the point where there aren't enough minutes to fill

out the physically allotted time for the entire conference.

Another possible higher standard would be to require that

the minutes be contributed by a relatively large number of PC members;

a simple way to enforce this would be to have a reduced limit for the

number of minutes any one PC member can throw at a PC member paper. If

a conference were to have slots of 10, 20 and 30 minutes, perhaps the

limit should be 5. Then a PC member paper would require the support of

two PC members to get a short slot, four to get a medium slot, and

six to get a long slot. (A subtle point to beware of is that,

under the usual system where a PC author must leave the room during

discussion of his paper, he still might be able to infer who else did or

did not support his paper if he has access to intermediate information about

who has how many minutes left. Therefore, if it is desired to maintain

the traditional secrecy about votes on PC papers,

it may be important to maintain secrecy about all votes,

casting them all at once at the end of the process, or at the end of each round of discussion. The PC chair will need to give some thought to how to manage the mechanics of the voting process so as to maintain secrecy. This all sounds like a good job for a computer. But will a PC member who fails to bring a laptop to the PC meeting be at a disadvantage?)

MF says
I don't submit papers to conferences where I serve on the PC.

I don't know what else to say.

--GuySteele

Here is a question that is independent of but somewhat related

the question of how much presentation time to give a paper:

how many pages of proceedings should be allotted?

MF says
I propose to allocate a smaller but equal number of pages

to all papers. If a proceedings currently consists of 200 pages, and

you accept 45 papers, give everyone 4 pages and then allocate the

remaining 20 to the papers that got "over the limit minutes" as a

prize.

- GuySteele

Guy asked excellent questions. I gave direct answers prefixed with

MF says
inside of Guy's post, which I hope to be a better way of

communicating in this case.

Note: As a former econ/business major, I am aware of "Macro 101"

and the rules of markets and the assumptions behind markets. Like

every other market, this one will also be imperfect initially. Time will

improve it. -- MatthiasFelleisen

Inasmuch as OOPSLA is a de-facto journal, reducing the paper page counts substantially must reduce the prestige of the technical stream.

Alternatively - we could end up like physics: the same content as our existing 10-page papers compressed into 4 pages. (Ok, on reflection, perhaps that would not be such a bad thing). An addendum (perhaps distributed as a CD to technical attendees and in the ACM DL only) could take any amount of 4-pagers - 40 SmallTalks, 30 IntruigingTechnologies, and 30 Posters, without reducing the tech stream at all, and giving us around 100 other presenters, so say 50 extra insider attendees.

But I still think any rejigging of the the technical programme may gives us another hundred attendees, if we are lucky. To do more, we need to think more broadly

JamesKjx

I'm not so sure, if reducing page size solves the problem. I've always wondered why having a paper accepted has always also the meaning that the paper will be presented. Especially all these academic papers, I think they're quite interesting in reading, but I would like to hear more directly usable talks. And for those maybe just the slides (probably online only) would be sufficient. Thus they wouldn't eat up space in the proceedings.

--JuttaEckstein

Yeah well, I would like more usable talks too. If OOPSLA is still technically a "conference" the technical papers should still be presented - and there are a hundred or so people who come to hear them - what's the problem with that?

It can sit happily on the side as one of many OOPSLA events - like the CHI & GRAPH techincal sessions - but it should not dominate the conference, it doesn't need the biggest rooms, and certainly shouldn't dominate website and advertising!

Then, there's lots of space and time - and in the procedings or an online onlyDL supplement or whatever - that could be used for other interestinguseful things that support wider communities.

Interesting question: how long would the tech programme keep running if it had only one 200 seater room & the proceedings appeared only online/in the DL.

What if, in addition, every presenters had to pay an extra 1000 USD to contribute to the room rental & PC costs?

My answer is - as long as ECOOP keeps running, as long as academic interest is strong, at least another 10 years?

OK, so again, the TechnicalProgramme is not the biggest problem. What do we do with the rest of the conference? Rebuild it on the model of SPA+JAOO?

JamesKjx