Logo di OpenPEC

Posta Elettronica Certificata (Certified E-Mail)

- Open Source Project -

Logo di Ksolutions S.p.A. Logo di EXEntrica S.r.l. Get OpenPEC at SourceForge.net. Fast, secure and Free Open Source software downloads Versione italiana
 Home > Documentation > Test
Law
:: The law
:: CNIPA website
 
Technical documentation
:: OpenPEC overview
:: Installing
:: Frequently Asked Questions (FAQ)
:: Tests

Seminars
:: Perl WorkShop 2004 - Flavio Fanton (pdf - html)
:: SALPA 2004 - Luca De Santis (pdf)
:: SALPA 2004 Call for paper - Luca De Santis (pdf)
:: Bologna 2004 - Giovanni Faglioni (pdf)
:: Formia 2005 - Flavio Fanton (pdf)
:: Governance e open source (Pisa 2006) - Umberto Ferrara (pdf)
:: ICT & Security (Pisa 2007) - Umberto Ferrara (pdf)
:: Convegno OS e riuso (Univ. Pisa 2008) - Umberto Ferrara (pdf)
:: Convegno TOSSlab (Navacchio 2008) - Umberto Ferrara (pdf)
:: Seminario TOSSlab (Univ. Pisa 2009) - Umberto Ferrara (pdf1 pdf2)
 
 

Servers

opec1 opec2
  • model name : 1 CPU Pentium III (Katmai)
  • cpu MHz : 501.148
  • cache size : 512 KB
  • RAM Memory : 192 MB
  • Hard Disk EIDE FUJITSU MPC3102AT E - 256K Cache 5400rpm UATA
  • model name : 1 CPU Pentium III (Katmai)
  • cpu MHz : 547.626
  • cache size : 512 KB
  • RAM Memory : 256 MB
  • Hard Disk EIDE 10GB WDC WD100BA - 2MB Cache 7200rpm UDMA/66
performance (source: hdparm)
  • Timing buffer-cache reads: 568 MB in 2.00 seconds = 284.00 MB/sec
  • Timing buffered disk reads: 38 MB in 3.10 seconds = 12.26 MB/sec
  • Timing buffer-cache reads: 580 MB in 2.01 seconds = 288.56 MB/sec
  • Timing buffered disk reads: 72 MB in 3.05 seconds = 23.61 MB/sec
 

Load test - session 1

The goal of this test is to assess the maximum load tolerated by OpenPec (not by the system on the whole) in therms of number of processed emails per time unit as a function of the processes number

In this first test session we chose to keed separated signing cases (ingoing mail from the client to delivery to the external world) from the verification cases ("waybill" emails to be delivered locally to the LMTP servers) in order to understand the incidence of each case.

Procedures for test execution
We installed the complete system (Postfix - OpenPec - Cyrus) on two servers, opec1 and opec2, then we used two additional servers to send emails. To ease the sending operations we configured Postfixs in order to relay also on the non secure SMTP.
We supplied every mailer (we used mailer.pl, included in the distribution) with 18 various 22525Kb messages with modified header (From and To fields) in order to be conformant to the envelop.
For every test we tested the number of processes for both Openpec and Postfix (transport).

Network architecture:

Software architecture on every server:

Sistema di PEC ad un solo domain certificato

Description

  1. Both servers must have Postfix up with empty queues and OpenPec down.
  2. Send emails to opec2.
  3. Start OpenPec on opec2, then execute the Postfix messages queue flush (postfix -f).
  4. Observe every parameter (cpu, execution time, etc.).
  5. Execute the point number 3 for opec1.
  6. Observe opec1 parameters.

Results

opec2 - signing + receipts
Processes Time
1 Test completion in 46'' -  Average: 2,6 sec/mail
2 Test completion in 46'' -  Average: 2,6 sec/mail
4 Test completion in 46'' -  Average: 2,6 sec/mail
8 Test completion in 46'' -  Average: 2,6 sec/mail
16 Test completion in 46'' -  Average: 2,6 sec/mail
opec1 - verifica + ricevute
Processes Time
1 Test completion in 3'55'' -  Average: 14 sec/mail
2 Test completion in 4'46'' secondi -  Average: 15 sec/mail
4 Test completion in 4'46'' secondi -  Average: 15 sec/mail
8 Test completion in 4'50'' -  Average: 15 sec/mail
16 -

Conclusions
Opec1 has execution time much greater than Opec2. This confirms that verification operation is more expensive than signing, almost an order of magnitude (even if we have to note that opec1 hard disk is worse than opec2 one).
However, the most important fact is that the more efficient configuration is with only one process per server, on monoprocessor servers.

Remarks

  • How much does the signing weigh on the average elaboration time per email?
  • We made test again, after modifying the code in order to make multiple calls (not only one) to openssl, and we obtained:
    the only signing test lasted 39'' (instead of 46''); the verification test lasted 1' 42'' (instead of 4' 46'').
  • How important is the hard drive?
    We made the test againg, exchanging the servers (since opec2 hard drive is more performant) and the result is a difference of roughly a quarter of the time:
    the signing test lasted 32'' (instead of 46'').
    This means that the hard drive choice is fundamental for the perfomances enhancement.

 

Load test - session 2

The goal is to esteem the maximum load tolerated by a complete model of a PEC system using OpenPec, in therms of number of processed emails per time unit.

Procedures for test execution
We installed the complete system (Postfix - OpenPec - Cyrus) on two servers, opec1 and opec2, and we used a third server to send emails. To ease the sending operations we configured Postfixs in order to relay also on the non secure SMTP.
The mass sending of emails has been done by two instances of the simply emailer shipped with the distribution (mailer.pl), one for each server. A set of various emails (with From, To and Cc fields modified to be conformant to the envelop) has been delivered to each instance. In particular:

  • opec1
    domain: domaincert1.openpec.org
    local users:
    user1@domaincert1.openpec.org
    user2@domaincert1.openpec.org

  • opec2
    domain: domaincert2.openpec.org
    local users:
    user1@domaincert2.openpec.org
    user2@domaincert2.openpec.org

  • third server
    • mailer 1:
      794 messages - 44.216KB total
      cmd: mailer.pl user1@domaincert1.openpec.org user2@domaincert2.openpec.org path_mail_1 1 opec1
    • mailer 2:
      1226 messages - 96.968KB total
      cmd: mailer.pl user1@domaincert2.openpec.org user2@domaincert1.openpec.org path_mail_2 1 opec2

Results

Processed messages group by kind of elaboration

  • opec1 - total time - 161' 34"
    • 96.968KB of verified messages (waybills from Opec2)
    • 193.936KB of signed messages (acknoledgement of token charge and acknoledgement of receipt to Opec2)
    • 44.226KB of signed messages (acceptance to Opec1)

  • opec2 - total time - 151' 03"
    • 44.226KB of verified messages (waybills from Opec1)
    • 88.452KB of signed messages (acknoledgement of token charge and acknoledgement of receipt to Opec1)
    • 96.968KB of signed messages (acceptance to Opec2)

Total processed/generated messages

  • opec1
    • 794 - 44.226KB acknoledgements of receipt
    • 794 - 44.226KB waybills generated
    • 1226 - 96.968KB waybills received
    • 1226 - 96.968KB acknoledgement of token charge
    • 1226 - 96.968KB acknoledgement of receipt
    • total processed: 5266 - 379.336KB

  • opec2 - total time - 151' 03"
    • 1226 - 96.968KB acceptance receipts
    • 1226 - 96.968KB waybills generated
    • 794 - 44.226KB waybills received
    • 794 - 44.226KB acknoledgement of token charge
    • 794 - 44.226KB acknoledgement of receipt
    • total processed: 4834 - 326.604KB

 

Reliability test

The goal is to put the system through a consistent load and analyse users' test mailboxes in order to verify the links between every different email typologies.
An email from PEC1 server to PEC2 server means:
a waybill
an acceptance receipt
an acknoledgement of token charge
an acknoledgement of receipt

Procedures for test execution
We installed the complete system (Postfix - OpenPec - Cyrus) on two servers, opec1 and opec2, and we used a third server to send emails and download mailboxes via pop3 from both servers. To ease the sending operations we configured Postfixs in order to relay also on the non secure SMTP.
The mass sending of emails has been done by two instances of the simply emailer shipped with the distribution (mailer.pl) with a dead time between each sending going from one to 15 sec, one for each server. A set of various emails (with From, To and Cc fields modified to be conformant to the envelop) has been delivered to each instance. In particular:

  • opec1
    domain: domaincert1.openpec.org
    local users:
    user1@domaincert1.openpec.org
    user2@domaincert1.openpec.org

  • opec2
    domain: domaincert2.openpec.org
    local users:
    user1@domaincert2.openpec.org
    user2@domaincert2.openpec.org

  • third server
    • mailer 1:
      794 messages - 44.216KB total
      cmd: mailer.pl user1@domaincert1.openpec.org user2@domaincert2.openpec.org path_mail_1 1 opec1
    • mailer 2:
      1226 messages - 96.968KB total
      cmd: mailer.pl user1@domaincert2.openpec.org user2@domaincert1.openpec.org path_mail_2 1 opec2

Results

Users received messages group by kind

  • opec1
    794 acknoledgements of acceptance
    794 acknoledgements of token charge
    794 acknoledgements of receipt
    1225 waybills

  • opec2
    1225 acknoledgements of acceptance
    1225 acknoledgements of token charge
    1225 acknoledgements of receipt
    794 waybills
    1 Undeliverable Mail

* The "Undeliverable Mail" message was caused by a not conformant header, which has been managed according to the specifications.

** The test was done using openssl version 0.9.7d.

*** Note that both OpenPec servers were configured with the maximum log level and they both produced a 250Mb log file.

Remarks
The test had previoulsy been done using openssl 0.9.7a and it had shown some problems (not particularly weighty): an incostant number (about 20) of transport anomalies were generated, on a sample of 1226 emails. This was caused by the failure of the waybill verification, even if the waybill was correctly signed. We analysed the matter deeper and we made sure that the failure was wrong: the error code number 11 was returned, without any code or message to STDERR or STDOUT; several tests demonstrated the problem randomness.
The problem isn't (weighty), becase the sender didn't receive any acknoledgement of receipt, so in the PEC context, he knew that the email hadn't been delivered.
The problem was solved updating to openssl version 0.9.7d

 

Load test - session 3

This test is the "load test -session 1" copy, but done on a multiprocessor server, in order to quantify the advantage with a more expensive solution and eventually show hypothetical code improvements.

Procedures for test execution
We installed the complete system (Postfix - OpenPec - Cyrus) on two servers, opec2 and opec3, then we used two additional servers to send emails. To ease the sending operations we configured Postfixs in order to relay also on the non secure SMTP.
We supplied every mailer (we used mailer.pl, included in the distribution) with 18 various 22525Kb messages with modified header (From and To fields) in order to be conformant to the envelop.
For every test we modified the number of processes for both Openpec and Postfix (transport).

Server:
opec3
  • model name : 2 CPU Pentium III (Katmai)
  • cpu MHz : 499.192
  • cache size : 512 KB
  • RAM Memory : 384 MB
  • Hard Disk SCSI HP 9.10GB A 80-SA40
prestazioni (fonte: hdparm)
  • Timing buffer-cache reads: 512 MB in 2.00 seconds = 255.62 MB/sec
  • Timing buffered disk reads: 78 MB in 3.09 seconds = 25.24 MB/sec

Results
opec3 - signing + receipts
Processes Time
1 Test completion in 46'' -  Average: 2,6 sec/email
2 Test completion in 29'' -  Average: 1,6 sec/email
4 Test completion in 29'' -  Average: 1,6 sec/email
8 Test completion in 29'' -  Average: 1,6 sec/email
16 Test completion in 29'' -  Average: 1,6 sec/email
opec3 - verification + receipts
Processes Time
1 Test completion in 3'20'' -  Average: 11,11 sec/email
2 Test completion in 2'11'' seconds -  Average: 7,28 sec/email
4 Test completion in 2' seconds -  Average: 6,67 sec/email
8 Test completion in 1'55'' -  Average: 6,39 sec/email
16 Test completion in 1'55'' -  Average: 6,39 sec/email

Conclusions
As expected we had a performance enhancement (about 30%) switching from 1 to 2 processes (one per processor) and then it raises to a stability; a slight drift is to be remarked on the verification test
During the 8 processes test (then comes the 16 processes one) the command watch -n 1 "ps ax|grep opec" shown that no more than 6 processes are used contemporary.
At this point, considering potential delays with network components external to OpenPec (LDAP server, MTA and LMTP server), the best configuration seems to be the one following this rule:

in /etc/opec.conf
$maxServers = <(number of processors * 2) + x>

where x is an integer to be chosen considering the server characteristics, expecially its hard drives: the more it is performant, the higher is the value (tipically from 1 to 3).