Profile of Mail/SpamAssassin/Message.pm

Filename	/usr/local/lib/perl5/site_perl/Mail/SpamAssassin/Message.pm
Statements	Executed 899857 statements in 5.43s

Subroutines
Calls	P	F	Exclusive Time	Inclusive Time	Subroutine
235	1	1	2.34s	4.55s	Mail::SpamAssassin::Message::::newMail::SpamAssassin::Message::new
198	1	1	2.02s	2.34s	Mail::SpamAssassin::Message::_parse_multipart
114864	20	1	379ms	379ms	Mail::SpamAssassin::Message::CORE:match (opcode)
1175	3	1	155ms	24.9s	Mail::SpamAssassin::Message::get_body_text_array_common
705	1	1	146ms	146ms	Mail::SpamAssassin::Message::split_into_array_of_short_lines
235	1	1	101ms	122ms	Mail::SpamAssassin::Message::::finishMail::SpamAssassin::Message::finish
235	1	1	75.9ms	2.63s	Mail::SpamAssassin::Message::parse_body
427	1	1	63.2ms	209ms	Mail::SpamAssassin::Message::_parse_normal
8145	2	1	49.8ms	49.8ms	Mail::SpamAssassin::Message::CORE:subst (opcode)
1253	4	1	32.5ms	32.5ms	Mail::SpamAssassin::Message::CORE:regcomp (opcode)
235	1	1	26.7ms	27.7ms	Mail::SpamAssassin::Message::get_all_metadata
705	1	1	24.5ms	24.6s	Mail::SpamAssassin::Message::get_rendered_body_text_array
1884	2	1	22.4ms	22.4ms	Mail::SpamAssassin::Message::get_metadata
705	1	1	22.2ms	2.80s	Mail::SpamAssassin::Message::find_parts
688	2	2	21.2ms	3.86s	Mail::SpamAssassin::Message::receive_date
940	4	1	16.9ms	16.9ms	Mail::SpamAssassin::Message::put_metadata
470	2	2	8.92ms	3.75s	Mail::SpamAssassin::Message::extract_message_metadata
1	1	1	8.80ms	49.1ms	Mail::SpamAssassin::Message::::BEGIN@55Mail::SpamAssassin::Message::BEGIN@55
688	1	1	7.51ms	7.51ms	Mail::SpamAssassin::Message::get_pristine_body
235	1	1	6.34ms	17.4ms	Mail::SpamAssassin::Message::finish_metadata
1	1	1	4.63ms	9.16ms	Mail::SpamAssassin::Message::::BEGIN@49Mail::SpamAssassin::Message::BEGIN@49
235	1	1	4.47ms	121ms	Mail::SpamAssassin::Message::get_invisible_rendered_body_text_array
705	1	1	4.42ms	4.42ms	Mail::SpamAssassin::Message::::CORE:qrMail::SpamAssassin::Message::CORE:qr (opcode)
235	1	1	4.26ms	246ms	Mail::SpamAssassin::Message::get_visible_rendered_body_text_array
235	1	1	3.53ms	3.53ms	Mail::SpamAssassin::Message::get_pristine_header
16	1	1	3.26ms	3.26ms	Mail::SpamAssassin::Message::CORE:unlink (opcode)
80	1	1	1.43ms	1.43ms	Mail::SpamAssassin::Message::::DESTROYMail::SpamAssassin::Message::DESTROY
1	1	1	1.32ms	218ms	Mail::SpamAssassin::Message::::BEGIN@56Mail::SpamAssassin::Message::BEGIN@56
235	1	1	1.03ms	1.03ms	Mail::SpamAssassin::Message::CORE:sort (opcode)
16	1	1	174µs	174µs	Mail::SpamAssassin::Message::CORE:close (opcode)
1	1	1	44µs	58µs	Mail::SpamAssassin::Message::::BEGIN@45Mail::SpamAssassin::Message::BEGIN@45
1	1	1	28µs	103µs	Mail::SpamAssassin::Message::::BEGIN@47Mail::SpamAssassin::Message::BEGIN@47
1	1	1	24µs	67µs	Mail::SpamAssassin::Message::::BEGIN@46Mail::SpamAssassin::Message::BEGIN@46
1	1	1	23µs	95µs	Mail::SpamAssassin::Message::::BEGIN@60Mail::SpamAssassin::Message::BEGIN@60
1	1	1	23µs	23µs	Mail::SpamAssassin::Message::::BEGIN@54Mail::SpamAssassin::Message::BEGIN@54
1	1	1	22µs	791µs	Mail::SpamAssassin::Message::::BEGIN@57Mail::SpamAssassin::Message::BEGIN@57
1	1	1	20µs	184µs	Mail::SpamAssassin::Message::::BEGIN@58Mail::SpamAssassin::Message::BEGIN@58
0	0	0	0s	0s	Mail::SpamAssassin::Message::delete_metadata
0	0	0	0s	0s	Mail::SpamAssassin::Message::::get_bodyMail::SpamAssassin::Message::get_body
0	0	0	0s	0s	Mail::SpamAssassin::Message::get_decoded_body_text_array
0	0	0	0s	0s	Mail::SpamAssassin::Message::get_mbox_separator
0	0	0	0s	0s	Mail::SpamAssassin::Message::get_mimepart_digests
0	0	0	0s	0s	Mail::SpamAssassin::Message::get_pristine
0	0	0	0s	0s	Mail::SpamAssassin::Message::split_into_array_of_short_paragraphs

Call graph for these subroutines as a Graphviz dot language file.

Line	State ments	Time on line	Calls	Time in subs	Code
1					# <@LICENSE>
2					# Licensed to the Apache Software Foundation (ASF) under one or more
3					# contributor license agreements. See the NOTICE file distributed with
4					# this work for additional information regarding copyright ownership.
5					# The ASF licenses this file to you under the Apache License, Version 2.0
6					# (the "License"); you may not use this file except in compliance with
7					# the License. You may obtain a copy of the License at:
8					#
9					# http://www.apache.org/licenses/LICENSE-2.0
10					#
11					# Unless required by applicable law or agreed to in writing, software
12					# distributed under the License is distributed on an "AS IS" BASIS,
13					# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14					# See the License for the specific language governing permissions and
15					# limitations under the License.
16					# </@LICENSE>
17
18					=head1 NAME
19
20					Mail::SpamAssassin::Message - decode, render, and hold an RFC-2822 message
21
22					=head1 DESCRIPTION
23
24					This module encapsulates an email message and allows access to the various MIME
25					message parts and message metadata.
26
27					The message structure, after initiating a parse() cycle, looks like this:
28
29					Message object, also top-level node in Message::Node tree
30					\|
31					+---> Message::Node for other parts in MIME structure
32					\| \|---> [ more Message::Node parts ... ]
33					\| [ others ... ]
34					\|
35					+---> Message::Metadata object to hold metadata
36
37					=head1 PUBLIC METHODS
38
39					=over 4
40
41					=cut
42
43					package Mail::SpamAssassin::Message;
44
45	2	69µs	2	73µs	# spent 58µs (44+14) within Mail::SpamAssassin::Message::BEGIN@45 which was called: # once (44µs+14µs) by Mail::SpamAssassin::BEGIN@75 at line 45 use strict; # spent 58µs making 1 call to Mail::SpamAssassin::Message::BEGIN@45 # spent 14µs making 1 call to strict::import
46	2	62µs	2	110µs	# spent 67µs (24+43) within Mail::SpamAssassin::Message::BEGIN@46 which was called: # once (24µs+43µs) by Mail::SpamAssassin::BEGIN@75 at line 46 use warnings; # spent 67µs making 1 call to Mail::SpamAssassin::Message::BEGIN@46 # spent 43µs making 1 call to warnings::import
47	2	156µs	2	178µs	# spent 103µs (28+75) within Mail::SpamAssassin::Message::BEGIN@47 which was called: # once (28µs+75µs) by Mail::SpamAssassin::BEGIN@75 at line 47 use re 'taint'; # spent 103µs making 1 call to Mail::SpamAssassin::Message::BEGIN@47 # spent 75µs making 1 call to re::import
48
49					# spent 9.16ms (4.63+4.53) within Mail::SpamAssassin::Message::BEGIN@49 which was called: # once (4.63ms+4.53ms) by Mail::SpamAssassin::BEGIN@75 at line 52 BEGIN {
50	3	263µs	1	621µs	eval { require Digest::SHA; import Digest::SHA qw(sha1 sha1_hex); 1 } # spent 621µs making 1 call to Exporter::import
51	1	10µs			or do { require Digest::SHA1; import Digest::SHA1 qw(sha1 sha1_hex) }
52	1	75µs	1	9.16ms	} # spent 9.16ms making 1 call to Mail::SpamAssassin::Message::BEGIN@49
53
54	2	65µs	1	23µs	# spent 23µs within Mail::SpamAssassin::Message::BEGIN@54 which was called: # once (23µs+0s) by Mail::SpamAssassin::BEGIN@75 at line 54 use Mail::SpamAssassin; # spent 23µs making 1 call to Mail::SpamAssassin::Message::BEGIN@54
55	2	390µs	1	49.1ms	# spent 49.1ms (8.80+40.3) within Mail::SpamAssassin::Message::BEGIN@55 which was called: # once (8.80ms+40.3ms) by Mail::SpamAssassin::BEGIN@75 at line 55 use Mail::SpamAssassin::Message::Node; # spent 49.1ms making 1 call to Mail::SpamAssassin::Message::BEGIN@55
56	2	409µs	1	218ms	# spent 218ms (1.32+217) within Mail::SpamAssassin::Message::BEGIN@56 which was called: # once (1.32ms+217ms) by Mail::SpamAssassin::BEGIN@75 at line 56 use Mail::SpamAssassin::Message::Metadata; # spent 218ms making 1 call to Mail::SpamAssassin::Message::BEGIN@56
57	2	86µs	2	1.56ms	# spent 791µs (22+769) within Mail::SpamAssassin::Message::BEGIN@57 which was called: # once (22µs+769µs) by Mail::SpamAssassin::BEGIN@75 at line 57 use Mail::SpamAssassin::Constants qw(:sa); # spent 791µs making 1 call to Mail::SpamAssassin::Message::BEGIN@57 # spent 769µs making 1 call to Exporter::import
58	2	73µs	2	348µs	# spent 184µs (20+164) within Mail::SpamAssassin::Message::BEGIN@58 which was called: # once (20µs+164µs) by Mail::SpamAssassin::BEGIN@75 at line 58 use Mail::SpamAssassin::Logger; # spent 184µs making 1 call to Mail::SpamAssassin::Message::BEGIN@58 # spent 164µs making 1 call to Exporter::import
59
60	2	9.71ms	2	166µs	# spent 95µs (23+72) within Mail::SpamAssassin::Message::BEGIN@60 which was called: # once (23µs+72µs) by Mail::SpamAssassin::BEGIN@75 at line 60 use vars qw(@ISA); # spent 95µs making 1 call to Mail::SpamAssassin::Message::BEGIN@60 # spent 72µs making 1 call to vars::import
61
62	1	22µs			@ISA = qw(Mail::SpamAssassin::Message::Node);
63
64					# ---------------------------------------------------------------------------
65
66					=item new()
67
68					Creates a Mail::SpamAssassin::Message object. Takes a hash reference
69					as a parameter. The used hash key/value pairs are as follows:
70
71					C<message> is either undef (which will use STDIN), a scalar - a string
72					containing an entire message, a reference to such string, an array reference
73					of the message with one line per array element, or either a file glob
74					or an IO::File object which holds the entire contents of the message.
75
76					Note: The message is expected to generally be in RFC 2822 format, optionally
77					including an mbox message separator line (the "From " line) as the first line.
78
79					C<parse_now> specifies whether or not to create the MIME tree
80					at object-creation time or later as necessary.
81
82					The I<parse_now> option, by default, is set to false (0).
83					This allows SpamAssassin to not have to generate the tree of
84					Mail::SpamAssassin::Message::Node objects and their related data if the
85					tree is not going to be used. This is handy, for instance, when running
86					C<spamassassin -d>, which only needs the pristine header and body which
87					is always handled when the object is created.
88
89					C<subparse> specifies how many MIME recursion levels should be parsed.
90					Defaults to 20.
91
92					=cut
93
94					# month mappings (ripped from Util.pm)
95	1	14µs			my %MONTH = (jan => 1, feb => 2, mar => 3, apr => 4, may => 5, jun => 6,
96					jul => 7, aug => 8, sep => 9, oct => 10, nov => 11, dec => 12);
97
98					# day of week mapping (starting from zero)
99	1	4µs			my @DAY_OF_WEEK = qw/Sun Mon Tue Wed Thu Fri Sat/ ;
100
101					# spent 4.55s (2.34+2.22) within Mail::SpamAssassin::Message::new which was called 235 times, avg 19.4ms/call: # 235 times (2.34s+2.22s) by Mail::SpamAssassin::parse at line 551 of Mail/SpamAssassin.pm, avg 19.4ms/call sub new {
102	235	661µs			my $class = shift;
103	235	633µs			$class = ref($class) \|\| $class;
104
105	235	559µs			my($opts) = @_;
106	235	984µs			my $message = defined $opts->{'message'} ? $opts->{'message'} : \*STDIN;
107	235	806µs			my $parsenow = $opts->{'parsenow'} \|\| 0;
108	235	636µs			my $normalize = $opts->{'normalize'} \|\| 0;
109
110					# Specifies whether or not to parse message/rfc822 parts into its own tree.
111					# If the # > 0, it'll subparse, otherwise it won't. By default, do twenty
112					# levels deep.
113	235	771µs			my $subparse = defined $opts->{'subparse'} ? $opts->{'subparse'} : 20;
114
115	235	3.07ms	235	7.16ms	my $self = $class->SUPER::new({normalize=>$normalize}); # spent 7.16ms making 235 calls to Mail::SpamAssassin::Message::Node::new, avg 30µs/call
116
117	235	856µs			$self->{tmpfiles} = [];
118	235	814µs			$self->{pristine_headers} = '';
119	235	819µs			$self->{pristine_body} = '';
120	235	1.04ms			$self->{mime_boundary_state} = {};
121	235	755µs			$self->{line_ending} = "\012";
122	235	722µs			$self->{master_deadline} = $opts->{'master_deadline'};
123	235	644µs			$self->{suppl_attrib} = $opts->{'suppl_attrib'};
124
125	235	523µs			if ($self->{suppl_attrib}) { # caller-provided additional information
126					# pristine_body_length is currently used by an eval test check_body_length
127					# Possible To-Do: Base the length on the @message array later down?
128					if (defined $self->{suppl_attrib}{body_size}) {
129					# Optional info provided by a caller; should reflect the original
130					# message body size if provided, and as such it may differ from the
131					# $self->{pristine_body} size, e.g. when the caller passed a truncated
132					# message to SpamAssassin, or when counting line-endings differently.
133					$self->{pristine_body_length} = $self->{suppl_attrib}{body_size};
134					}
135					if (ref $self->{suppl_attrib}{mimepart_digests}) {
136					# Optional info provided by a caller: an array of digest codes (e.g. SHA1)
137					# of each MIME part. Should reflect the original message if provided.
138					# As such it may differ from digests calculated by get_mimepart_digests(),
139					# e.g. when the caller passed a truncated message to SpamAssassin.
140					$self->{mimepart_digests} = $self->{suppl_attrib}{mimepart_digests};
141					}
142					}
143
144	235	561µs			bless($self,$class);
145
146					# create the metadata holder class
147	235	2.81ms	235	4.53ms	$self->{metadata} = Mail::SpamAssassin::Message::Metadata->new($self); # spent 4.53ms making 235 calls to Mail::SpamAssassin::Message::Metadata::new, avg 19µs/call
148
149					# Ok, go ahead and do the message "parsing"
150
151					# protect it from abuse ...
152	235	451µs			local $_;
153
154					# Figure out how the message was passed to us, and deal with it.
155	235	477µs			my @message;
156	235	1.35ms			if (ref $message eq 'ARRAY') {
157	470	52.9ms			@message = @{$message};
158					}
159					elsif (ref($message) eq 'GLOB' \|\| ref($message) =~ /^IO::/) {
160					if (defined fileno $message) {
161
162					# sysread+split avoids a Perl I/O bug (Bug 5985)
163					# and is faster than (<$message>) by 10..25 %
164					# (a drawback is a short-term double storage of a text in $raw_str)
165					#
166					my($nread,$raw_str); $raw_str = '';
167					while ( $nread=sysread($message, $raw_str, 16384, length $raw_str) ) { }
168					defined $nread or die "error reading: $!";
169					@message = split(/^/m, $raw_str, -1);
170
171					if ($raw_str eq '') {
172					dbg("message: empty message read");
173					} elsif (length($raw_str) > 128*1024) {
174					# ditch rarely used large chunks of allocated memory, Bug 6514
175					# http://www.perlmonks.org/?node_id=803515
176					# about 97% of mail messages are below 128 kB,
177					# about 98% of mail messages are below 256 kB (2010 statistics)
178					# dbg("message: deallocating %.2f MB", length($raw_str)/1024/1024);
179					undef $raw_str;
180					}
181					}
182					}
183					elsif (ref $message eq 'SCALAR') {
184					@message = split(/^/m, $$message, -1);
185					}
186					elsif (ref $message) {
187					dbg("message: Input is a reference of unknown type!");
188					}
189					elsif (defined $message) {
190					@message = split(/^/m, $message, -1);
191					}
192
193					# Pull off mbox and mbx separators
194					# also deal with null messages
195	235	4.77ms	470	1.51ms	if (!@message) { # spent 1.51ms making 470 calls to Mail::SpamAssassin::Message::CORE:match, avg 3µs/call
196					# bug 4884:
197					# if we get here, it means that the input was null, so fake the message
198					# content as a single newline...
199					@message = ("\n");
200					} elsif ($message[0] =~ /^From\s+(?!:)/) {
201					# careful not to confuse with obsolete syntax which allowed WSP before ':'
202					# mbox formated mailbox
203					$self->{'mbox_sep'} = shift @message;
204					} elsif ($message[0] =~ MBX_SEPARATOR) {
205					$_ = shift @message;
206
207					# Munge the mbx message separator into mbox format as a sort of
208					# de facto portability standard in SA's internals. We need to
209					# to this so that Mail::SpamAssassin::Util::parse_rfc822_date
210					# can parse the date string...
211					if (/([\s\d]\d)-([a-zA-Z]{3})-(\d{4})\s(\d{2}):(\d{2}):(\d{2})/) {
212					# $1 = day of month
213					# $2 = month (text)
214					# $3 = year
215					# $4 = hour
216					# $5 = min
217					# $6 = sec
218					my @arr = localtime(timelocal($6,$5,$4,$1,$MONTH{lc($2)}-1,$3));
219					my $address;
220					foreach (@message) {
221					if (/^From:[^<]*<([^>]+)>/) {
222					$address = $1;
223					last;
224					} elsif (/^From:\s*([^<> ]+)/) {
225					$address = $1;
226					last;
227					}
228					}
229					$self->{'mbox_sep'} = "From $address $DAY_OF_WEEK[$arr[6]] $2 $1 $4:$5:$6 $3\n";
230					}
231					}
232
233					# bug 4363
234					# Check to see if we should do CRLF instead of just LF
235					# For now, just check the first and last line and do whatever it does
236	235	11.1ms	470	1.39ms	if (@message && ($message[0] =~ /\015\012/ \|\| $message[-1] =~ /\015\012/)) { # spent 1.39ms making 470 calls to Mail::SpamAssassin::Message::CORE:match, avg 3µs/call
237					$self->{line_ending} = "\015\012";
238					dbg("message: line ending changed to CRLF");
239					}
240
241					# Is a CRLF -> LF line endings conversion necessary?
242	235	940µs			my $squash_crlf = $self->{line_ending} eq "\015\012";
243
244					# Go through all the header fields of the message
245	235	487µs			my $hdr_errors = 0;
246	235	426µs			my $header;
247	235	432µs			for (;;) {
248					# make sure not to lose the last header field when there is no body
249	13561	42.3ms			my $eof = !@message;
250	13561	38.8ms			my $current = $eof ? "\n" : shift @message;
251
252	13561	218ms	13561	49.6ms	if ( $current =~ /^[ \t]/ ) { # spent 49.6ms making 13561 calls to Mail::SpamAssassin::Message::CORE:match, avg 4µs/call
253					# This wasn't useful in terms of a rule, but we may want to treat it
254					# specially at some point. Perhaps ignore it?
255					#unless ($current =~ /\S/) {
256					# $self->{'obsolete_folding_whitespace'} = 1;
257					#}
258
259	5886	10.0ms			$header = '' if !defined $header; # header starts with a continuation!?
260	5886	16.5ms			$header .= $current; # append continuations, no matter what
261	5886	38.1ms			$self->{'pristine_headers'} .= $current;
262					}
263					else { # not a continuation
264					# Ok, there's a header here, let's go ahead and add it in.
265	7675	26.6ms			if (defined $header) { # deal with a previous header field
266	7440	58.7ms			my ($key, $value) = split (/:/s, $header, 2);
267
268					# If it's not a valid header (aka: not in the form "foo:bar"), skip it.
269	7440	28.7ms			if (defined $value) {
270					# CRLF -> LF line-endings conversion if necessary
271	7440	11.7ms			$value =~ s/\015\012/\012/sg if $squash_crlf;
272	7440	98.1ms	7440	20.7ms	$key =~ s/[ \t]+\z//; # strip WSP before colon, obsolete rfc822 syn # spent 20.7ms making 7440 calls to Mail::SpamAssassin::Message::CORE:subst, avg 3µs/call
273					# limit the length of the pairs we store
274	7440	19.6ms			if (length($key) > MAX_HEADER_KEY_LENGTH) {
275					$key = substr($key, 0, MAX_HEADER_KEY_LENGTH);
276					$self->{'truncated_header'} = 1;
277					}
278	7440	14.1ms			if (length($value) > MAX_HEADER_VALUE_LENGTH) {
279					$value = substr($value, 0, MAX_HEADER_VALUE_LENGTH);
280					$self->{'truncated_header'} = 1;
281					}
282	7440	56.2ms	7440	1.77s	$self->header($key, $value); # spent 1.77s making 7440 calls to Mail::SpamAssassin::Message::Node::header, avg 237µs/call
283					}
284					}
285
286	7675	197ms	15115	42.2ms	if ($current =~ /^\r?$/) { # a regular end of a header section # spent 42.2ms making 15115 calls to Mail::SpamAssassin::Message::CORE:match, avg 3µs/call
287	235	879µs			if ($eof) {
288					$self->{'missing_head_body_separator'} = 1;
289					} else {
290	235	785µs			$self->{'pristine_headers'} .= $current;
291					}
292	235	728µs			last;
293					}
294					elsif ($current =~ /^--/) { # mime boundary encountered, bail out
295					$self->{'missing_head_body_separator'} = 1;
296					unshift(@message, $current);
297					last;
298					}
299					# should we assume entering a body on encountering invalid header field?
300					else {
301					# no re "strict"; # since perl 5.21.8: Ranges of ASCII printables...
302	7440	120ms	7440	38.1ms	if ($current !~ /^[\041-\071\073-\176]+[ \t]*:/) { # spent 38.1ms making 7440 calls to Mail::SpamAssassin::Message::CORE:match, avg 5µs/call
303					# A field name MUST be composed of printable US-ASCII characters
304					# (i.e., characters that have values between 33 (041) and 126 (176),
305					# inclusive), except colon (072). Obsolete header field syntax
306					# allowed WSP before a colon.
307					if (++$hdr_errors <= 3) {
308					# just consume but ignore a few invalid header fields
309					} else { # enough is enough...
310					$self->{'missing_head_body_separator'} = 1;
311					unshift(@message, $current);
312					last;
313					}
314					}
315					}
316
317					# start collecting a new header field
318	7440	16.0ms			$header = $current;
319	7440	48.8ms			$self->{'pristine_headers'} .= $current;
320					}
321					}
322	235	595µs			undef $header;
323
324					# Store the pristine body for later -- store as a copy since @message
325					# will get modified below
326	235	24.1ms			$self->{'pristine_body'} = join('', @message);
327
328	235	1.03ms			if (!defined $self->{pristine_body_length}) {
329	235	1.32ms			$self->{'pristine_body_length'} = length $self->{'pristine_body'};
330					}
331
332					# iterate over lines in reverse order
333					# merge multiple blank lines into a single one
334	235	470µs			my $start;
335	235	249ms			for (my $cnt=$#message; $cnt>=0; $cnt--) {
336					# CRLF -> LF line-endings conversion if necessary
337	71228	110ms			$message[$cnt] =~ s/\015\012\z/\012/ if $squash_crlf;
338
339					# line is blank
340	71228	893ms	71228	217ms	if ($message[$cnt] =~ /^\s*$/) { # spent 217ms making 71228 calls to Mail::SpamAssassin::Message::CORE:match, avg 3µs/call
341					# /^\s*$/ is about 5% faster then !/\S/, but still expensive here
342	4614	12.7ms			if (!defined $start) {
343	3360	5.73ms			$start=$cnt;
344					}
345	4614	9.21ms			next unless $cnt == 0;
346					}
347
348					# line is not blank, or we've reached the beginning
349
350					# if we've got a series of blank lines, get rid of them
351	66644	118ms			if (defined $start) {
352	3360	5.68ms			my $max_blank_lines = 20;
353	3360	5.87ms			my $num = $start-$cnt;
354	3360	5.47ms			if ($num > $max_blank_lines) {
355	6	37µs			splice @message, $cnt+2, $num-$max_blank_lines;
356					}
357	3360	29.7ms			undef $start;
358					}
359					}
360
361					# Figure out the boundary
362	235	458µs			my ($boundary);
363	235	5.27ms	470	54.9ms	($self->{'type'}, $boundary) = Mail::SpamAssassin::Util::parse_content_type($self->header('content-type')); # spent 34.0ms making 235 calls to Mail::SpamAssassin::Util::parse_content_type, avg 144µs/call # spent 21.0ms making 235 calls to Mail::SpamAssassin::Message::Node::header, avg 89µs/call
364	235	2.72ms	235	2.23ms	dbg("message: main message type: ".$self->{'type'}); # spent 2.23ms making 235 calls to Mail::SpamAssassin::Logger::dbg, avg 10µs/call
365
366					# dbg("message: \$message[0]: \"" . $message[0] . "\"");
367
368					# bug 6845: if main message type is multipart and the message body does not begin with
369					# either a blank line or the boundary (if defined), insert a blank line
370					# to ensure proper parsing - do not consider MIME headers at the beginning of the body
371					# to be part of the message headers.
372	235	5.15ms	429	1.77ms	if ($self->{'type'} =~ /^multipart\//i && $#message > 0 && $message[0] =~ /\S/) # spent 1.77ms making 429 calls to Mail::SpamAssassin::Message::CORE:match, avg 4µs/call
373					{
374	171	12.6ms	342	9.49ms	if (!defined $boundary \|\| $message[0] !~ /^--\Q$boundary\E/) # spent 8.91ms making 171 calls to Mail::SpamAssassin::Message::CORE:regcomp, avg 52µs/call # spent 576µs making 171 calls to Mail::SpamAssassin::Message::CORE:match, avg 3µs/call
375					{
376	40	264µs	40	282µs	dbg("message: Inserting blank line at top of body to ensure correct multipart MIME parsing"); # spent 282µs making 40 calls to Mail::SpamAssassin::Logger::dbg, avg 7µs/call
377	40	191µs			unshift(@message, "\012");
378					}
379					}
380
381					# dbg("message: \$message[0]: \"" . $message[0] . "\"");
382					# dbg("message: \$message[1]: \"" . $message[1] . "\"");
383
384					# parse queue, simple array of parts to parse:
385					# 0: part object, already in the tree
386					# 1: boundary used to focus body parsing
387					# 2: message content
388					# 3: how many MIME subparts to parse down
389					#
390	235	1.87ms			$self->{'parse_queue'} = [ [ $self, $boundary, \@message, $subparse ] ];
391
392					# If the message does need to get parsed, save off a copy of the body
393					# in a format we can easily parse later so we don't have to rip from
394					# pristine_body ... If we do want to parse now, go ahead and do so ...
395					#
396	235	450µs			if ($parsenow) {
397					$self->parse_body();
398					}
399
400	235	4.71ms			$self;
401					}
402
403					# ---------------------------------------------------------------------------
404
405					=item find_parts()
406
407					Used to search the tree for specific MIME parts. See
408					I<Mail::SpamAssassin::Message::Node> for more details.
409
410					=cut
411
412					# Used to find any MIME parts whose simple content-type matches a given regexp
413					# Searches it's own and any children parts. Returns an array of MIME
414					# objects which match.
415					#
416					# spent 2.80s (22.2ms+2.78) within Mail::SpamAssassin::Message::find_parts which was called 705 times, avg 3.97ms/call: # 705 times (22.2ms+2.78s) by Mail::SpamAssassin::Message::get_body_text_array_common at line 1111, avg 3.97ms/call sub find_parts {
417	705	1.77ms			my $self = shift;
418
419					# ok, we need to do the parsing now...
420	705	3.42ms	235	2.63s	$self->parse_body() if (exists $self->{'parse_queue'}); # spent 2.63s making 235 calls to Mail::SpamAssassin::Message::parse_body, avg 11.2ms/call
421
422					# and pass through to the Message::Node version of the method
423	705	18.4ms	705	148ms	return $self->SUPER::find_parts(@_); # spent 148ms making 705 calls to Mail::SpamAssassin::Message::Node::find_parts, avg 209µs/call
424					}
425
426					# ---------------------------------------------------------------------------
427
428					=item get_pristine_header()
429
430					Returns pristine headers of the message. If no specific header name
431					is given as a parameter (case-insensitive), then all headers will be
432					returned as a scalar, including the blank line at the end of the headers.
433
434					If called in an array context, an array will be returned with each
435					specific header in a different element. In a scalar context, the last
436					specific header is returned.
437
438					ie: If 'Subject' is specified as the header, and there are 2 Subject
439					headers in a message, the last/bottom one in the message is returned in
440					scalar context or both are returned in array context.
441
442					Btw, returning the last header field (not the first) happens to be consistent
443					with DKIM signatures, which search for and cover multiple header fields
444					bottom-up according to the 'h' tag. Let's keep it this way.
445
446					Note: the returned header will include the ending newline and any embedded
447					whitespace folding.
448
449					=cut
450
451					# spent 3.53ms within Mail::SpamAssassin::Message::get_pristine_header which was called 235 times, avg 15µs/call: # 235 times (3.53ms+0s) by Mail::SpamAssassin::PerMsgStatus::_get at line 1913 of Mail/SpamAssassin/PerMsgStatus.pm, avg 15µs/call sub get_pristine_header {
452	235	543µs			my ($self, $hdr) = @_;
453
454	235	3.39ms			return $self->{pristine_headers} if !defined $hdr \|\| $hdr eq '';
455					my(@ret) =
456					$self->{pristine_headers} =~ /^\Q$hdr\E[ \t]:[ \t](.*?\n(?![ \t]))/smgi;
457					# taintedness is retained by "use re 'taint'" (fix in bug 5283 now redundant)
458					if (!@ret) {
459					return $self->get_header($hdr);
460					} elsif (wantarray) {
461					return @ret;
462					} else {
463					return $ret[-1];
464					}
465					}
466
467					=item get_mbox_separator()
468
469					Returns the mbox separator found in the message, or undef if there
470					wasn't one.
471
472					=cut
473
474					sub get_mbox_separator {
475					return $_[0]->{mbox_sep};
476					}
477
478					=item get_body()
479
480					Returns an array of the pristine message body, one line per array element.
481
482					=cut
483
484					sub get_body {
485					my ($self) = @_;
486					my @ret = split(/^/m, $self->{pristine_body});
487					return \@ret;
488					}
489
490					# ---------------------------------------------------------------------------
491
492					=item get_pristine()
493
494					Returns a scalar of the entire pristine message.
495
496					=cut
497
498					sub get_pristine {
499					my ($self) = @_;
500					return $self->{pristine_headers} . $self->{pristine_body};
501					}
502
503					=item get_pristine_body()
504
505					Returns a scalar of the pristine message body.
506
507					=cut
508
509					# spent 7.51ms within Mail::SpamAssassin::Message::get_pristine_body which was called 688 times, avg 11µs/call: # 688 times (7.51ms+0s) by Mail::SpamAssassin::Plugin::Bayes::get_msgid at line 998 of Mail/SpamAssassin/Plugin/Bayes.pm, avg 11µs/call sub get_pristine_body {
510	688	1.52ms			my ($self) = @_;
511	688	7.48ms			return $self->{pristine_body};
512					}
513
514					# ---------------------------------------------------------------------------
515
516					=item extract_message_metadata($permsgstatus)
517
518					=cut
519
520					# spent 3.75s (8.92ms+3.74) within Mail::SpamAssassin::Message::extract_message_metadata which was called 470 times, avg 7.97ms/call: # 235 times (6.48ms+3.74s) by Mail::SpamAssassin::PerMsgStatus::extract_message_metadata at line 1703 of Mail/SpamAssassin/PerMsgStatus.pm, avg 15.9ms/call # 235 times (2.44ms+0s) by Mail::SpamAssassin::Plugin::Bayes::get_body_from_msg at line 1025 of Mail/SpamAssassin/Plugin/Bayes.pm, avg 10µs/call sub extract_message_metadata {
521	470	1.01ms			my ($self, $permsgstatus) = @_;
522
523					# do this only once per message, it can be expensive
524	470	2.73ms			return if $self->{already_extracted_metadata};
525	235	1.22ms			$self->{already_extracted_metadata} = 1;
526
527	235	3.71ms	235	3.74s	$self->{metadata}->extract ($self, $permsgstatus); # spent 3.74s making 235 calls to Mail::SpamAssassin::Message::Metadata::extract, avg 15.9ms/call
528					}
529
530					# ---------------------------------------------------------------------------
531
532					=item $str = get_metadata($hdr)
533
534					=cut
535
536					# spent 22.4ms within Mail::SpamAssassin::Message::get_metadata which was called 1884 times, avg 12µs/call: # 1649 times (19.8ms+0s) by Mail::SpamAssassin::PerMsgStatus::_get at line 1987 of Mail/SpamAssassin/PerMsgStatus.pm, avg 12µs/call # 235 times (2.65ms+0s) by Mail::SpamAssassin::PerMsgStatus::extract_message_metadata at line 1741 of Mail/SpamAssassin/PerMsgStatus.pm, avg 11µs/call sub get_metadata {
537	1884	4.35ms			my ($self, $hdr) = @_;
538	1884	4.02ms			if (!$self->{metadata}) {
539					warn "metadata: oops! get_metadata() called after finish_metadata()"; return;
540					}
541					# dbg("message: get_metadata - %s: %s", $hdr, defined $_ ? $_ : '<undef>')
542					# for $self->{metadata}->{strings}->{lc $hdr};
543
544	1884	20.4ms			$self->{metadata}->{strings}->{lc $hdr};
545					}
546
547					=item put_metadata($hdr, $text)
548
549					=cut
550
551					# spent 16.9ms within Mail::SpamAssassin::Message::put_metadata which was called 940 times, avg 18µs/call: # 235 times (4.59ms+0s) by Mail::SpamAssassin::Message::Metadata::parse_received_headers at line 284 of Mail/SpamAssassin/Message/Metadata/Received.pm, avg 20µs/call # 235 times (4.32ms+0s) by Mail::SpamAssassin::Message::Metadata::parse_received_headers at line 280 of Mail/SpamAssassin/Message/Metadata/Received.pm, avg 18µs/call # 235 times (4.17ms+0s) by Mail::SpamAssassin::Message::Metadata::parse_received_headers at line 282 of Mail/SpamAssassin/Message/Metadata/Received.pm, avg 18µs/call # 235 times (3.86ms+0s) by Mail::SpamAssassin::Message::Metadata::parse_received_headers at line 286 of Mail/SpamAssassin/Message/Metadata/Received.pm, avg 16µs/call sub put_metadata {
552	940	3.87ms			my ($self, $hdr, $text) = @_;
553	940	1.96ms			if (!$self->{metadata}) {
554					warn "metadata: oops! put_metadata() called after finish_metadata()"; return;
555					}
556					# dbg("message: put_metadata - %s: %s", $hdr, $text);
557	940	13.2ms			$self->{metadata}->{strings}->{lc $hdr} = $text;
558					}
559
560					=item delete_metadata($hdr)
561
562					=cut
563
564					sub delete_metadata {
565					my ($self, $hdr) = @_;
566					if (!$self->{metadata}) {
567					warn "metadata: oops! delete_metadata() called after finish_metadata()"; return;
568					}
569					delete $self->{metadata}->{strings}->{lc $hdr};
570					}
571
572					=item $str = get_all_metadata()
573
574					=cut
575
576					# spent 27.7ms (26.7+1.03) within Mail::SpamAssassin::Message::get_all_metadata which was called 235 times, avg 118µs/call: # 235 times (26.7ms+1.03ms) by Mail::SpamAssassin::Plugin::Bayes::_tokenize_headers at line 1304 of Mail/SpamAssassin/Plugin/Bayes.pm, avg 118µs/call sub get_all_metadata {
577	235	602µs			my ($self) = @_;
578
579	235	798µs			if (!$self->{metadata}) {
580					warn "metadata: oops! get_all_metadata() called after finish_metadata()"; return;
581					}
582	235	444µs			my @ret;
583	235	858µs			my $keys_ref = $self->{metadata}->{strings};
584	235	4.63ms	235	1.03ms	foreach my $key (sort keys %$keys_ref) { # spent 1.03ms making 235 calls to Mail::SpamAssassin::Message::CORE:sort, avg 4µs/call
585	940	5.86ms			my $val = $keys_ref->{$key};
586	940	1.64ms			$val = '' if !defined $val;
587	940	9.85ms			push (@ret, "$key: $val\n");
588					}
589	235	3.35ms			return (wantarray ? @ret : join('', @ret));
590					}
591
592					# ---------------------------------------------------------------------------
593
594					=item finish_metadata()
595
596					Destroys the metadata for this message. Once a message has been
597					scanned fully, the metadata is no longer required. Destroying
598					this will free up some memory.
599
600					=cut
601
602					# spent 17.4ms (6.34+11.0) within Mail::SpamAssassin::Message::finish_metadata which was called 235 times, avg 74µs/call: # 235 times (6.34ms+11.0ms) by Mail::SpamAssassin::Message::finish at line 620, avg 74µs/call sub finish_metadata {
603	235	520µs			my ($self) = @_;
604	235	2.48ms			if (defined ($self->{metadata})) {
605	235	2.54ms	235	11.0ms	$self->{metadata}->finish(); # spent 11.0ms making 235 calls to Mail::SpamAssassin::Message::Metadata::finish, avg 47µs/call
606	235	815µs			delete $self->{metadata};
607					}
608					}
609
610					=item finish()
611
612					Clean up an object so that it can be destroyed.
613
614					=cut
615
616					# spent 122ms (101+20.8) within Mail::SpamAssassin::Message::finish which was called 235 times, avg 519µs/call: # 235 times (101ms+20.8ms) by main::wanted at line 588 of /usr/local/bin/sa-learn, avg 519µs/call sub finish {
617	235	541µs			my ($self) = @_;
618
619					# Clean ourself up
620	235	2.03ms	235	17.4ms	$self->finish_metadata(); # spent 17.4ms making 235 calls to Mail::SpamAssassin::Message::finish_metadata, avg 74µs/call
621
622					# These will only be in the root Message node
623	235	1.13ms			delete $self->{'mime_boundary_state'};
624	235	547µs			delete $self->{'mbox_sep'};
625	235	627µs			delete $self->{'normalize'};
626	235	1.05ms			delete $self->{'pristine_body'};
627	235	910µs			delete $self->{'pristine_headers'};
628	235	725µs			delete $self->{'line_ending'};
629	235	572µs			delete $self->{'missing_head_body_separator'};
630
631	235	1.03ms			my @toclean = ( $self );
632
633					# Go ahead and clean up all of the Message::Node parts
634	235	7.22ms			while (my $part = shift @toclean) {
635					# bug 5557: windows requires tmp file be closed before it can be rm'd
636	625	2.39ms			if (ref $part->{'raw'} eq 'GLOB') {
637	16	359µs	16	174µs	close($part->{'raw'}) or die "error closing input file: $!"; # spent 174µs making 16 calls to Mail::SpamAssassin::Message::CORE:close, avg 11µs/call
638					}
639
640					# bug 5858: avoid memory leak with deep MIME structure
641	625	1.29ms			if (defined ($part->{metadata})) {
642					$part->{metadata}->finish();
643					delete $part->{metadata};
644					}
645
646	625	12.0ms			delete $part->{'headers'};
647	625	13.2ms			delete $part->{'raw_headers'};
648	625	5.63ms			delete $part->{'header_order'};
649	625	31.4ms			delete $part->{'raw'};
650	625	1.83ms			delete $part->{'decoded'};
651	625	1.54ms			delete $part->{'rendered'};
652	625	1.64ms			delete $part->{'visible_rendered'};
653	625	1.38ms			delete $part->{'invisible_rendered'};
654	625	1.71ms			delete $part->{'type'};
655	625	1.29ms			delete $part->{'rendered_type'};
656
657					# if there are children nodes, add them to the queue of nodes to clean up
658	625	1.69ms			if (exists $part->{'body_parts'}) {
659	396	1.53ms			push(@toclean, @{$part->{'body_parts'}});
660	198	536µs			delete $part->{'body_parts'};
661					}
662					}
663
664					# delete temporary files
665	235	2.76ms			if ($self->{'tmpfiles'}) {
666	470	2.04ms			for my $fn (@{$self->{'tmpfiles'}}) {
667	16	3.47ms	16	3.26ms	unlink($fn) or warn "cannot unlink $fn: $!"; # spent 3.26ms making 16 calls to Mail::SpamAssassin::Message::CORE:unlink, avg 203µs/call
668					}
669	235	655µs			delete $self->{'tmpfiles'};
670					}
671					}
672
673					# also use a DESTROY method, just to ensure (as much as possible) that
674					# temporary files are deleted even if the finish() method is omitted
675					# spent 1.43ms within Mail::SpamAssassin::Message::DESTROY which was called 80 times, avg 18µs/call: # 80 times (1.43ms+0s) by main::wanted at line 589 of /usr/local/bin/sa-learn, avg 18µs/call sub DESTROY {
676	80	177µs			my $self = shift;
677					# best practices: prevent potential calls to eval and to system routines
678					# in code of a DESTROY method from clobbering global variables $@ and $!
679	80	513µs			local($@,$!); # keep outer error handling unaffected by DESTROY
680	80	895µs			if ($self->{'tmpfiles'}) {
681					for my $fn (@{$self->{'tmpfiles'}}) {
682					unlink($fn) or dbg("message: cannot unlink $fn: $!");
683					}
684					}
685					}
686
687					# ---------------------------------------------------------------------------
688
689					=item receive_date()
690
691					Return a time_t value with the received date of the current message,
692					or current time if received time couldn't be determined.
693
694					=cut
695
696					# spent 3.86s (21.2ms+3.84) within Mail::SpamAssassin::Message::receive_date which was called 688 times, avg 5.61ms/call: # 453 times (13.8ms+2.51s) by Mail::SpamAssassin::Plugin::TxRep::check_senders_reputation at line 1249 of Mail/SpamAssassin/Plugin/TxRep.pm, avg 5.56ms/call # 235 times (7.44ms+1.33s) by Mail::SpamAssassin::Plugin::Bayes::_learn_trapped at line 465 of Mail/SpamAssassin/Plugin/Bayes.pm, avg 5.69ms/call sub receive_date {
697	688	1.65ms			my($self) = @_;
698
699	688	17.5ms	1376	3.84s	return Mail::SpamAssassin::Util::receive_date(scalar $self->get_all_headers(0,1)); # spent 3.41s making 688 calls to Mail::SpamAssassin::Message::Node::get_all_headers, avg 4.96ms/call # spent 427ms making 688 calls to Mail::SpamAssassin::Util::receive_date, avg 621µs/call
700					}
701
702					# ---------------------------------------------------------------------------
703
704					=back
705
706					=head1 PARSING METHODS, NON-PUBLIC
707
708					These methods take a RFC2822-esque formatted message and create a tree
709					with all of the MIME body parts included. Those parts will be decoded
710					as necessary, and text/html parts will be rendered into a standard text
711					format, suitable for use in SpamAssassin.
712
713					=over 4
714
715					=item parse_body()
716
717					parse_body() passes the body part that was passed in onto the
718					correct part parser, either _parse_multipart() for multipart/* parts,
719					or _parse_normal() for everything else. Multipart sections become the
720					root of sub-trees, while everything else becomes a leaf in the tree.
721
722					For multipart messages, the first call to parse_body() doesn't create a
723					new sub-tree and just uses the parent node to contain children. All other
724					calls to parse_body() will cause a new sub-tree root to be created and
725					children will exist underneath that root. (this is just so the tree
726					doesn't have a root node which points at the actual root node ...)
727
728					=cut
729
730					# spent 2.63s (75.9ms+2.55) within Mail::SpamAssassin::Message::parse_body which was called 235 times, avg 11.2ms/call: # 235 times (75.9ms+2.55s) by Mail::SpamAssassin::Message::find_parts at line 420, avg 11.2ms/call sub parse_body {
731	235	552µs			my($self) = @_;
732
733					# This shouldn't happen, but just in case, abort.
734	235	618µs			return unless (exists $self->{'parse_queue'});
735
736	235	1.57ms	235	1.47ms	dbg("message: ---- MIME PARSER START ----"); # spent 1.47ms making 235 calls to Mail::SpamAssassin::Logger::dbg, avg 6µs/call
737
738	1095	32.3ms			while (my $toparse = shift @{$self->{'parse_queue'}}) {
739					# multipart sections are required to have a boundary set ... If this
740					# one doesn't, assume it's malformed and send it to be parsed as a
741					# non-multipart section
742					#
743	625	2.51ms			my ($msg, $boundary, $body, $subparse) = @$toparse;
744
745	625	22.4ms	625	2.36ms	if ($msg->{'type'} =~ m{^multipart/}i && defined $boundary && $subparse > 0) { # spent 2.36ms making 625 calls to Mail::SpamAssassin::Message::CORE:match, avg 4µs/call
746	198	1.82ms	198	2.34s	$self->_parse_multipart($toparse); # spent 2.34s making 198 calls to Mail::SpamAssassin::Message::_parse_multipart, avg 11.8ms/call
747					}
748					else {
749					# If it's not multipart, go ahead and just deal with it.
750	427	3.29ms	427	209ms	$self->_parse_normal($toparse); # spent 209ms making 427 calls to Mail::SpamAssassin::Message::_parse_normal, avg 490µs/call
751
752					# bug 5041: process message/*, but exclude message/partial content types
753	427	4.69ms	427	1.33ms	if ($msg->{'type'} =~ m{^message/(?!partial\z)}i && $subparse > 0) # spent 1.33ms making 427 calls to Mail::SpamAssassin::Message::CORE:match, avg 3µs/call
754					{
755					# Just decode the part, but we don't need the resulting string here.
756					$msg->decode(0);
757
758					# bug 7125: decode and parse only message/rfc822 or message/global,
759					# but do not treat other message/* content types (like the ones listed
760					# here) as a message consisting of a header and a body, as they are not:
761					# message/delivery-status, message/global-delivery-status,
762					# message/feedback-report, message/global-headers,
763					# message/global-disposition-notification,
764					# message/disposition-notification, (and message/partial)
765
766					# bug 5051, bug 3748: check $msg->{decoded}: sometimes message/* parts
767					# have no content, and we get stuck waiting for STDIN, which is bad. :(
768
769					if ($msg->{'type'} =~ m{^message/(?:rfc822\|global)\z}i &&
770					defined $msg->{'decoded'} && $msg->{'decoded'} ne '')
771					{
772					# Ok, so this part is still semi-recursive, since M::SA::Message
773					# calls M::SA::Message, but we don't subparse the new message,
774					# and pull a sneaky "steal our child's queue" maneuver to deal
775					# with it on our own time. Reference the decoded array directly
776					# since it's faster.
777					#
778					my $msg_obj = Mail::SpamAssassin::Message->new({
779					message => $msg->{'decoded'},
780					parsenow => 0,
781					normalize => $self->{normalize},
782					subparse => $subparse - 1,
783					});
784
785					# Add the new message to the current node
786					$msg->add_body_part($msg_obj);
787
788					# now this is the sneaky bit ... steal the sub-message's parse_queue
789					# and add it to ours. then we'll handle the sub-message in our
790					# normal loop and get all the glory. muhaha. :)
791					push(@{$self->{'parse_queue'}}, @{$msg_obj->{'parse_queue'}});
792					delete $msg_obj->{'parse_queue'};
793
794					# Ok, we've subparsed, so go ahead and remove the raw and decoded
795					# data because we won't need them anymore (the tree under this part
796					# will have that data)
797					if (ref $msg->{'raw'} eq 'GLOB') {
798					# Make sure we close it if it's a temp file -- Bug 5166
799					close($msg->{'raw'})
800					or die "error closing input file: $!";
801					}
802
803					delete $msg->{'raw'};
804
805					delete $msg->{'decoded'};
806					}
807					}
808					}
809					}
810
811	235	1.65ms	235	1.39ms	dbg("message: ---- MIME PARSER END ----"); # spent 1.39ms making 235 calls to Mail::SpamAssassin::Logger::dbg, avg 6µs/call
812
813					# we're done parsing, so remove the queue variable
814	235	2.45ms			delete $self->{'parse_queue'};
815					}
816
817					=item _parse_multipart()
818
819					Generate a root node, and for each child part call parse_body()
820					to generate the tree.
821
822					=cut
823
824					# spent 2.34s (2.02+321ms) within Mail::SpamAssassin::Message::_parse_multipart which was called 198 times, avg 11.8ms/call: # 198 times (2.02s+321ms) by Mail::SpamAssassin::Message::parse_body at line 746, avg 11.8ms/call sub _parse_multipart {
825	198	415µs			my($self, $toparse) = @_;
826
827	396	1.48ms			my ($msg, $boundary, $body, $subparse) = @{$toparse};
828
829					# we're not supposed to be a leaf, so prep ourselves
830	198	946µs			$msg->{'body_parts'} = [];
831
832					# the next set of objects will be one level deeper
833	198	407µs			$subparse--;
834
835	198	1.99ms	198	1.38ms	dbg("message: parsing multipart, got boundary: ".(defined $boundary ? $boundary : '')); # spent 1.38ms making 198 calls to Mail::SpamAssassin::Logger::dbg, avg 7µs/call
836
837					# NOTE: The MIME boundary REs here are very specific to be mostly RFC 1521
838					# compliant, but also allow possible malformations to still work. Please
839					# see Bugzilla bug 3749 for more information before making any changes!
840
841					# ignore preamble per RFC 1521, unless there's no boundary ...
842	198	915µs			if ( defined $boundary ) {
843	198	355µs			my $line;
844	396	1.20ms			my $tmp_line = @{$body};
845	198	1.50ms			for ($line=0; $line < $tmp_line; $line++) {
846					# dbg("message: multipart line $line: \"" . $body->[$line] . "\"");
847					# specifically look for an opening boundary
848	340	14.1ms	396	8.62ms	if (substr($body->[$line],0,2) eq '--' # triage # spent 7.78ms making 198 calls to Mail::SpamAssassin::Message::CORE:regcomp, avg 39µs/call # spent 839µs making 198 calls to Mail::SpamAssassin::Message::CORE:match, avg 4µs/call
849					&& $body->[$line] =~ /^--\Q$boundary\E\s*$/) {
850					# Make note that we found the opening boundary
851	198	990µs			$self->{mime_boundary_state}->{$boundary} = 1;
852
853					# if the line after the opening boundary isn't a header, flag it.
854					# we need to make sure that there's actually another line though.
855					# no re "strict"; # since perl 5.21.8: Ranges of ASCII printables...
856	198	3.16ms	198	1.10ms	if ($line+1 < $tmp_line && $body->[$line+1] !~ /^[\041-\071\073-\176]+:/) { # spent 1.10ms making 198 calls to Mail::SpamAssassin::Message::CORE:match, avg 6µs/call
857					$self->{'missing_mime_headers'} = 1;
858					}
859
860	198	764µs			last;
861					}
862					}
863
864					# Found a boundary, ignore the preamble
865	198	802µs			if ( $line < $tmp_line ) {
866	396	1.70ms			splice @{$body}, 0, $line+1;
867					}
868
869					# Else, there's no boundary, so leave the whole part...
870					}
871
872					# prepare a new tree node
873	198	3.27ms	198	5.24ms	my $part_msg = Mail::SpamAssassin::Message::Node->new({ normalize=>$self->{normalize} }); # spent 5.24ms making 198 calls to Mail::SpamAssassin::Message::Node::new, avg 26µs/call
874	198	449µs			my $in_body = 0;
875	198	383µs			my $header;
876					my $part_array;
877					my $found_end_boundary;
878
879	396	1.20ms			my $line_count = @{$body};
880	396	6.03ms			foreach ( @{$body} ) {
881					# if we're on the last body line, or we find any boundary marker,
882					# deal with the mime part;
883					# a triage before an unlikely-to-match regexp avoids a CPU hotspot
884	67595	357ms	988	11.6ms	$found_end_boundary = defined $boundary && substr($_,0,2) eq '--' # spent 9.11ms making 494 calls to Mail::SpamAssassin::Message::CORE:regcomp, avg 18µs/call # spent 2.44ms making 494 calls to Mail::SpamAssassin::Message::CORE:match, avg 5µs/call
885					&& /^--\Q$boundary\E(?:--)?\s*$/;
886	67595	110ms			if ( --$line_count == 0 \|\| $found_end_boundary ) {
887	390	1.60ms			my $line = $_; # remember the last line
888
889					# If at last line and no end boundary found, the line belongs to body
890					# TODO:
891					# Is $self->{mime_boundary_state}->{$boundary}-- needed here?
892					# Could "missing end boundary" be a useful rule? Mark it somewhere?
893					# If SA processed truncated message from amavis etc, this could also
894					# be hit legimately..
895	390	1.96ms			if (!$found_end_boundary) {
896					# TODO: This is duplicate code from few pages down below..
897					while (length ($_) > MAX_BODY_LINE_LENGTH) {
898					push (@{$part_array}, substr($_, 0, MAX_BODY_LINE_LENGTH)."\n");
899					substr($_, 0, MAX_BODY_LINE_LENGTH) = '';
900					}
901					push ( @{$part_array}, $_ );
902					}
903					# per rfc 1521, the CRLF before the boundary is part of the boundary:
904					# NOTE: The CRLF preceding the encapsulation line is conceptually
905					# attached to the boundary so that it is possible to have a part
906					# that does not end with a CRLF (line break). Body parts that must
907					# be considered to end with line breaks, therefore, must have two
908					# CRLFs preceding the encapsulation line, the first of which is part
909					# of the preceding body part, and the second of which is part of the
910					# encapsulation boundary.
911					elsif ($part_array) {
912	390	1.69ms			chomp( $part_array->[-1] ); # trim the CRLF that's part of the boundary
913	685	2.53ms			splice @{$part_array}, -1 if ( $part_array->[-1] eq '' ); # blank line for the boundary only ...
914					}
915					else {
916					# Invalid parts can have no body, so fake in a blank body
917					# in that case.
918					$part_array = [];
919					}
920
921	390	675µs			my($p_boundary);
922	390	7.78ms	780	96.1ms	($part_msg->{'type'}, $p_boundary) = Mail::SpamAssassin::Util::parse_content_type($part_msg->header('content-type')); # spent 65.7ms making 390 calls to Mail::SpamAssassin::Util::parse_content_type, avg 169µs/call # spent 30.3ms making 390 calls to Mail::SpamAssassin::Message::Node::header, avg 78µs/call
923	390	1.01ms			$p_boundary \|\|= $boundary;
924	390	4.53ms	390	3.15ms	dbg("message: found part of type ".$part_msg->{'type'}.", boundary: ".(defined $p_boundary ? $p_boundary : '')); # spent 3.15ms making 390 calls to Mail::SpamAssassin::Logger::dbg, avg 8µs/call
925
926					# we've created a new node object, so add it to the queue along with the
927					# text that belongs to that part, then add the new part to the current
928					# node to create the tree.
929	780	4.95ms			push(@{$self->{'parse_queue'}}, [ $part_msg, $p_boundary, $part_array, $subparse ]);
930	390	3.13ms	390	14.8ms	$msg->add_body_part($part_msg); # spent 14.8ms making 390 calls to Mail::SpamAssassin::Message::Node::add_body_part, avg 38µs/call
931
932					# rfc 1521 says /^--boundary--$/, some MUAs may just require /^--boundary--/
933					# but this causes problems with horizontal lines when the boundary is
934					# made up of dashes as well, etc.
935	390	1.26ms			if (defined $boundary) {
936					# no re "strict"; # since perl 5.21.8: Ranges of ASCII printables...
937	390	24.0ms	972	9.02ms	if ($line =~ /^--\Q${boundary}\E--\s*$/) { # spent 6.70ms making 390 calls to Mail::SpamAssassin::Message::CORE:regcomp, avg 17µs/call # spent 2.33ms making 582 calls to Mail::SpamAssassin::Message::CORE:match, avg 4µs/call
938					# Make a note that we've seen the end boundary
939	198	623µs			$self->{mime_boundary_state}->{$boundary}--;
940	198	998µs			last;
941					}
942					elsif ($line_count && $body->[-$line_count] !~ /^[\041-\071\073-\176]+:/) {
943					# if we aren't on an end boundary and there are still lines left, it
944					# means we hit a new start boundary. therefore, the next line ought
945					# to be a mime header. if it's not, mark it.
946					$self->{'missing_mime_headers'} = 1;
947					}
948					}
949
950					# make sure we start with a new clean node
951	192	420µs			$in_body = 0;
952	192	2.11ms	192	5.59ms	$part_msg = Mail::SpamAssassin::Message::Node->new({ normalize=>$self->{normalize} }); # spent 5.59ms making 192 calls to Mail::SpamAssassin::Message::Node::new, avg 29µs/call
953	192	419µs			undef $part_array;
954	192	482µs			undef $header;
955
956	192	737µs			next;
957					}
958
959	67205	104ms			if (!$in_body) {
960					# s/\s+$//; # bug 5127: don't clean this up (yet)
961					# no re "strict"; # since perl 5.21.8: Ranges of ASCII printables...
962	1241	31.3ms	1697	8.27ms	if (/^[\041-\071\073-\176]+[ \t]*:/) { # spent 8.27ms making 1697 calls to Mail::SpamAssassin::Message::CORE:match, avg 5µs/call
963	785	3.12ms			if ($header) {
964	395	4.71ms			my ( $key, $value ) = split ( /:\s*/, $header, 2 );
965	395	3.07ms	395	78.1ms	$part_msg->header( $key, $value ); # spent 78.1ms making 395 calls to Mail::SpamAssassin::Message::Node::header, avg 198µs/call
966					}
967	785	2.04ms			$header = $_;
968	785	1.69ms			next;
969					}
970					elsif (/^[ \t]/ && $header) {
971					# $_ =~ s/^\s*//; # bug 5127, again
972	66	289µs			$header .= $_;
973	66	166µs			next;
974					}
975					else {
976	390	1.69ms			if ($header) {
977	390	4.35ms			my ( $key, $value ) = split ( /:\s*/, $header, 2 );
978	390	2.83ms	390	75.9ms	$part_msg->header( $key, $value ); # spent 75.9ms making 390 calls to Mail::SpamAssassin::Message::Node::header, avg 195µs/call
979					}
980	390	1.81ms			$in_body = 1;
981
982					# if there's a blank line separator, that's good. if there isn't,
983					# it's a body line, so drop through.
984	390	4.87ms	390	1.98ms	if (/^\r?$/) { # spent 1.98ms making 390 calls to Mail::SpamAssassin::Message::CORE:match, avg 5µs/call
985	390	898µs			next;
986					}
987					else {
988					$self->{'missing_mime_head_body_separator'} = 1;
989					}
990					}
991					}
992
993					# we run into a perl bug if the lines are astronomically long (probably
994					# due to lots of regexp backtracking); so split any individual line
995					# over MAX_BODY_LINE_LENGTH bytes in length. This can wreck HTML
996					# totally -- but IMHO the only reason a luser would use
997					# MAX_BODY_LINE_LENGTH-byte lines is to crash filters, anyway.
998	65964	290ms			while (length ($_) > MAX_BODY_LINE_LENGTH) {
999					push (@{$part_array}, substr($_, 0, MAX_BODY_LINE_LENGTH)."\n");
1000					substr($_, 0, MAX_BODY_LINE_LENGTH) = '';
1001					}
1002	131928	997ms			push ( @{$part_array}, $_ );
1003					}
1004
1005					# Look for a message epilogue
1006					# originally ignored whitespace: 0.185 0.2037 0.0654 0.757 0.00 0.00 TVD_TAB
1007					# ham FPs were all "." on a line by itself.
1008					# spams seem to only have NULL chars afterwards ?
1009	198	2.02ms			if ($line_count) {
1010	76	742µs			for(; $line_count > 0; $line_count--) {
1011	88	1.03ms	88	342µs	if ($body->[-$line_count] =~ /[^\s.]/) { # spent 342µs making 88 calls to Mail::SpamAssassin::Message::CORE:match, avg 4µs/call
1012					$self->{mime_epilogue_exists} = 1;
1013					last;
1014					}
1015					}
1016					}
1017
1018					}
1019
1020					=item _parse_normal()
1021
1022					Generate a leaf node and add it to the parent.
1023
1024					=cut
1025
1026					# spent 209ms (63.2+146) within Mail::SpamAssassin::Message::_parse_normal which was called 427 times, avg 490µs/call: # 427 times (63.2ms+146ms) by Mail::SpamAssassin::Message::parse_body at line 750, avg 490µs/call sub _parse_normal {
1027	427	871µs			my($self, $toparse) = @_;
1028
1029	854	3.00ms			my ($msg, $boundary, $body) = @{$toparse};
1030
1031	427	2.70ms	427	2.60ms	dbg("message: parsing normal part"); # spent 2.60ms making 427 calls to Mail::SpamAssassin::Logger::dbg, avg 6µs/call
1032
1033					# 0: content-type, 1: boundary, 2: charset, 3: filename
1034	427	6.93ms	854	93.3ms	my @ct = Mail::SpamAssassin::Util::parse_content_type($msg->header('content-type')); # spent 70.2ms making 427 calls to Mail::SpamAssassin::Util::parse_content_type, avg 164µs/call # spent 23.1ms making 427 calls to Mail::SpamAssassin::Message::Node::header, avg 54µs/call
1035
1036					# multipart sections are required to have a boundary set ... If this
1037					# one doesn't, assume it's malformed and revert to text/plain
1038	427	5.11ms	427	1.19ms	$msg->{'type'} = ($ct[0] !~ m@^multipart/@i \|\| defined $boundary ) ? $ct[0] : 'text/plain'; # spent 1.19ms making 427 calls to Mail::SpamAssassin::Message::CORE:match, avg 3µs/call
1039	427	2.05ms			$msg->{'charset'} = $ct[2];
1040
1041					# attempt to figure out a name for this attachment if there is one ...
1042	427	3.23ms	427	22.0ms	my $disp = $msg->header('content-disposition') \|\| ''; # spent 22.0ms making 427 calls to Mail::SpamAssassin::Message::Node::header, avg 52µs/call
1043	427	4.28ms	427	933µs	if ($disp =~ /name="?([^\";]+)"?/i) { # spent 933µs making 427 calls to Mail::SpamAssassin::Message::CORE:match, avg 2µs/call
1044	9	51µs			$msg->{'name'} = $1;
1045					}
1046					elsif ($ct[3]) {
1047	6	29µs			$msg->{'name'} = $ct[3];
1048					}
1049
1050	427	2.31ms			$msg->{'boundary'} = $boundary;
1051
1052					# If the part type is not one that we're likely to want to use, go
1053					# ahead and write the part data out to a temp file -- why keep sucking
1054					# up RAM with something we're not going to use?
1055					#
1056	427	6.65ms	427	3.49ms	if ($msg->{'type'} !~ m@^(?:text/(?:plain\|html)$\|message\b)@) { # spent 3.49ms making 427 calls to Mail::SpamAssassin::Message::CORE:match, avg 8µs/call
1057	16	31µs			my($filepath, $fh);
1058					eval {
1059	32	188µs	16	11.0ms	($filepath, $fh) = Mail::SpamAssassin::Util::secure_tmpfile(); 1; # spent 11.0ms making 16 calls to Mail::SpamAssassin::Util::secure_tmpfile, avg 689µs/call
1060	16	76µs			} or do {
1061					my $eval_stat = $@ ne '' ? $@ : "errno=$!"; chomp $eval_stat;
1062					info("message: failed to create a temp file: %s", $eval_stat);
1063					};
1064	16	66µs			if ($fh) {
1065					# The temp file was created, add it to the list of pending deletions
1066					# we cannot just delete immediately in the POSIX idiom, as this is
1067					# unportable (to win32 at least)
1068	32	138µs			push @{$self->{tmpfiles}}, $filepath;
1069	16	108µs	16	93µs	dbg("message: storing a message part to file %s", $filepath); # spent 93µs making 16 calls to Mail::SpamAssassin::Logger::dbg, avg 6µs/call
1070	32	462µs	16	7.95ms	$fh->print(@{$body}) or die "error writing to $filepath: $!"; # spent 7.95ms making 16 calls to IO::Handle::print, avg 497µs/call
1071	16	1.12ms	16	956µs	$fh->flush or die "error writing (flush) to $filepath: $!"; # spent 956µs making 16 calls to IO::Handle::flush, avg 60µs/call
1072	16	85µs			$msg->{'raw'} = $fh;
1073					}
1074					}
1075
1076					# if the part didn't get a temp file, go ahead and store the data in memory
1077	427	6.18ms			if (!defined $msg->{'raw'}) {
1078	411	2.62ms	411	2.55ms	dbg("message: storing a body to memory"); # spent 2.55ms making 411 calls to Mail::SpamAssassin::Logger::dbg, avg 6µs/call
1079	411	2.64ms			$msg->{'raw'} = $body;
1080					}
1081					}
1082
1083					# ---------------------------------------------------------------------------
1084
1085					sub get_mimepart_digests {
1086					my ($self) = @_;
1087
1088					if (!exists $self->{mimepart_digests}) {
1089					# traverse all parts which are leaves, recursively
1090					$self->{mimepart_digests} =
1091					[ map(sha1_hex($_->decode) . ':' . lc($_->{type}\|\|''),
1092					$self->find_parts(qr/^/,1,1)) ];
1093					}
1094					return $self->{mimepart_digests};
1095					}
1096
1097					# ---------------------------------------------------------------------------
1098
1099					# common code for get_rendered_body_text_array,
1100					# get_visible_rendered_body_text_array, get_invisible_rendered_body_text_array
1101					#
1102					# spent 24.9s (155ms+24.8) within Mail::SpamAssassin::Message::get_body_text_array_common which was called 1175 times, avg 21.2ms/call: # 705 times (70.0ms+24.5s) by Mail::SpamAssassin::Message::get_rendered_body_text_array at line 1159, avg 34.9ms/call # 235 times (51.8ms+189ms) by Mail::SpamAssassin::Message::get_visible_rendered_body_text_array at line 1164, avg 1.03ms/call # 235 times (33.3ms+83.7ms) by Mail::SpamAssassin::Message::get_invisible_rendered_body_text_array at line 1169, avg 498µs/call sub get_body_text_array_common {
1103	1175	3.40ms			my ($self, $method_name) = @_;
1104
1105	1175	3.58ms			my $key = 'text_' . $method_name;
1106	1645	9.56ms			if (exists $self->{$key}) { return $self->{$key} }
1107
1108	705	2.55ms			$self->{$key} = [];
1109
1110					# Find all parts which are leaves
1111	705	17.3ms	1410	2.80s	my @parts = $self->find_parts(qr/./,1); # spent 2.80s making 705 calls to Mail::SpamAssassin::Message::find_parts, avg 3.97ms/call # spent 4.42ms making 705 calls to Mail::SpamAssassin::Message::CORE:qr, avg 6µs/call
1112	705	1.56ms			return $self->{$key} unless @parts;
1113
1114					# the html metadata may have already been set, so let's not bother if it's
1115					# already been done.
1116	705	2.57ms			my $html_needs_setting = !exists $self->{metadata}->{html};
1117
1118	705	7.72ms	470	51.3ms	my $text = $method_name eq 'invisible_rendered' ? '' # spent 51.3ms making 470 calls to Mail::SpamAssassin::Message::Node::get_header, avg 109µs/call
1119					: ($self->get_header('subject') \|\| "\n");
1120
1121					# Go through each part
1122	705	12.7ms			for (my $pt = 0 ; $pt <= $#parts ; $pt++ ) {
1123	1281	2.89ms			my $p = $parts[$pt];
1124
1125					# put a blank line between parts ...
1126	1281	3.77ms			$text .= "\n" if $text ne '';
1127
1128	1281	14.3ms	1281	21.7s	my($type, $rnd) = $p->$method_name(); # decode this part # spent 21.7s making 427 calls to Mail::SpamAssassin::Message::Node::rendered, avg 50.9ms/call # spent 15.9ms making 427 calls to Mail::SpamAssassin::Message::Node::visible_rendered, avg 37µs/call # spent 14.6ms making 427 calls to Mail::SpamAssassin::Message::Node::invisible_rendered, avg 34µs/call
1129	1281	5.32ms			if ( defined $rnd ) {
1130					# Only text/* types are rendered ...
1131	1233	6.53ms			$text .= $rnd;
1132
1133					# TVD - if there are multiple parts, what should we do?
1134					# right now, just use the last one. we may need to give some priority
1135					# at some point, ie: use text/html rendered if it exists, or
1136					# text/plain rendered as html otherwise.
1137	1233	2.93ms			if ($html_needs_setting && $type eq 'text/html') {
1138	190	810µs			$self->{metadata}->{html} = $p->{html_results};
1139					}
1140					}
1141					}
1142
1143					# whitespace handling (warning: small changes have large effects!)
1144	705	36.1ms	705	29.2ms	$text =~ s/\n+\s*\n+/\f/gs; # double newlines => form feed # spent 29.2ms making 705 calls to Mail::SpamAssassin::Message::CORE:subst, avg 41µs/call
1145					# $text =~ tr/ \t\n\r\x0b\xa0/ /s; # whitespace (incl. VT, NBSP) => space
1146	705	11.9ms			$text =~ tr/ \t\n\r\x0b/ /s; # whitespace (incl. VT) => space
1147	705	5.21ms			$text =~ tr/\f/\n/; # form feeds => newline
1148
1149	705	13.4ms	705	146ms	my @textary = split_into_array_of_short_lines($text); # spent 146ms making 705 calls to Mail::SpamAssassin::Message::split_into_array_of_short_lines, avg 207µs/call
1150	705	3.49ms			$self->{$key} = \@textary;
1151
1152	705	21.1ms			return $self->{$key};
1153					}
1154
1155					# ---------------------------------------------------------------------------
1156
1157					# spent 24.6s (24.5ms+24.6) within Mail::SpamAssassin::Message::get_rendered_body_text_array which was called 705 times, avg 34.9ms/call: # 705 times (24.5ms+24.6s) by Mail::SpamAssassin::PerMsgStatus::get_decoded_stripped_body_text_array at line 1785 of Mail/SpamAssassin/PerMsgStatus.pm, avg 34.9ms/call sub get_rendered_body_text_array {
1158	705	1.44ms			my ($self) = @_;
1159	705	10.0ms	705	24.6s	return $self->get_body_text_array_common('rendered'); # spent 24.6s making 705 calls to Mail::SpamAssassin::Message::get_body_text_array_common, avg 34.9ms/call
1160					}
1161
1162					# spent 246ms (4.26+241) within Mail::SpamAssassin::Message::get_visible_rendered_body_text_array which was called 235 times, avg 1.04ms/call: # 235 times (4.26ms+241ms) by Mail::SpamAssassin::Plugin::Bayes::_get_msgdata_from_permsgstatus at line 1044 of Mail/SpamAssassin/Plugin/Bayes.pm, avg 1.04ms/call sub get_visible_rendered_body_text_array {
1163	235	554µs			my ($self) = @_;
1164	235	3.81ms	235	241ms	return $self->get_body_text_array_common('visible_rendered'); # spent 241ms making 235 calls to Mail::SpamAssassin::Message::get_body_text_array_common, avg 1.03ms/call
1165					}
1166
1167					# spent 121ms (4.47+117) within Mail::SpamAssassin::Message::get_invisible_rendered_body_text_array which was called 235 times, avg 517µs/call: # 235 times (4.47ms+117ms) by Mail::SpamAssassin::Plugin::Bayes::_get_msgdata_from_permsgstatus at line 1046 of Mail/SpamAssassin/Plugin/Bayes.pm, avg 517µs/call sub get_invisible_rendered_body_text_array {
1168	235	528µs			my ($self) = @_;
1169	235	2.83ms	235	117ms	return $self->get_body_text_array_common('invisible_rendered'); # spent 117ms making 235 calls to Mail::SpamAssassin::Message::get_body_text_array_common, avg 498µs/call
1170					}
1171
1172					# ---------------------------------------------------------------------------
1173
1174					sub get_decoded_body_text_array {
1175					my ($self) = @_;
1176
1177					if (defined $self->{text_decoded}) { return $self->{text_decoded}; }
1178					$self->{text_decoded} = [ ];
1179
1180					# Find all parts which are leaves
1181					my @parts = $self->find_parts(qr/^(?:text\|message)\b/i,1);
1182					return $self->{text_decoded} unless @parts;
1183
1184					# Go through each part
1185					for(my $pt = 0 ; $pt <= $#parts ; $pt++ ) {
1186					# bug 4843: skip text/calendar parts since they're usually an attachment
1187					# and not displayed
1188					next if ($parts[$pt]->{'type'} eq 'text/calendar');
1189
1190					push(@{$self->{text_decoded}}, "\n") if ( @{$self->{text_decoded}} );
1191					push(@{$self->{text_decoded}},
1192					split_into_array_of_short_paragraphs($parts[$pt]->decode()));
1193					}
1194
1195					return $self->{text_decoded};
1196					}
1197
1198					# ---------------------------------------------------------------------------
1199
1200					# spent 146ms within Mail::SpamAssassin::Message::split_into_array_of_short_lines which was called 705 times, avg 207µs/call: # 705 times (146ms+0s) by Mail::SpamAssassin::Message::get_body_text_array_common at line 1149, avg 207µs/call sub split_into_array_of_short_lines {
1201	705	1.37ms			my @result;
1202	705	17.5ms			foreach my $line (split (/^/m, $_[0])) {
1203	8927	32.8ms			while (length ($line) > MAX_BODY_LINE_LENGTH) {
1204					# try splitting "nicely" so that we don't chop a url in half or
1205					# something. if there's no space, then just split at max length.
1206	92	1.05ms			my $length = rindex($line, ' ', MAX_BODY_LINE_LENGTH) + 1;
1207	92	182µs			$length \|\|= MAX_BODY_LINE_LENGTH;
1208	92	1.14ms			push (@result, substr($line, 0, $length, ''));
1209					}
1210	8927	80.6ms			push (@result, $line);
1211					}
1212	705	14.6ms			@result;
1213					}
1214
1215					# ---------------------------------------------------------------------------
1216
1217					# split a text into array of paragraphs of sizes between
1218					# $chunk_size and 2 * $chunk_size, returning the resulting array
1219
1220					sub split_into_array_of_short_paragraphs {
1221					my @result;
1222					my $chunk_size = 1024;
1223					my $text_l = length($_[0]);
1224					my($j,$ofs);
1225					for ($ofs = 0; $text_l - $ofs > 2 * $chunk_size; $ofs = $j+1) {
1226					$j = index($_[0], "\n", $ofs+$chunk_size);
1227					if ($j < 0) {
1228					$j = index($_[0], " ", $ofs+$chunk_size);
1229					if ($j < 0) { $j = $ofs+$chunk_size }
1230					}
1231					push(@result, substr($_[0], $ofs, $j-$ofs+1));
1232					}
1233					push(@result, substr($_[0], $ofs)) if $ofs < $text_l;
1234					@result;
1235					}
1236
1237					# ---------------------------------------------------------------------------
1238
1239	1	20µs			1;
1240
1241					=back
1242
1243					=cut

					# spent 174µs within Mail::SpamAssassin::Message::CORE:close which was called 16 times, avg 11µs/call: # 16 times (174µs+0s) by Mail::SpamAssassin::Message::finish at line 637, avg 11µs/call sub Mail::SpamAssassin::Message::CORE:close; # opcode
					# spent 379ms within Mail::SpamAssassin::Message::CORE:match which was called 114864 times, avg 3µs/call: # 71228 times (217ms+0s) by Mail::SpamAssassin::Message::new at line 340, avg 3µs/call # 15115 times (42.2ms+0s) by Mail::SpamAssassin::Message::new at line 286, avg 3µs/call # 13561 times (49.6ms+0s) by Mail::SpamAssassin::Message::new at line 252, avg 4µs/call # 7440 times (38.1ms+0s) by Mail::SpamAssassin::Message::new at line 302, avg 5µs/call # 1697 times (8.27ms+0s) by Mail::SpamAssassin::Message::_parse_multipart at line 962, avg 5µs/call # 625 times (2.36ms+0s) by Mail::SpamAssassin::Message::parse_body at line 745, avg 4µs/call # 582 times (2.33ms+0s) by Mail::SpamAssassin::Message::_parse_multipart at line 937, avg 4µs/call # 494 times (2.44ms+0s) by Mail::SpamAssassin::Message::_parse_multipart at line 884, avg 5µs/call # 470 times (1.51ms+0s) by Mail::SpamAssassin::Message::new at line 195, avg 3µs/call # 470 times (1.39ms+0s) by Mail::SpamAssassin::Message::new at line 236, avg 3µs/call # 429 times (1.77ms+0s) by Mail::SpamAssassin::Message::new at line 372, avg 4µs/call # 427 times (3.49ms+0s) by Mail::SpamAssassin::Message::_parse_normal at line 1056, avg 8µs/call # 427 times (1.33ms+0s) by Mail::SpamAssassin::Message::parse_body at line 753, avg 3µs/call # 427 times (1.19ms+0s) by Mail::SpamAssassin::Message::_parse_normal at line 1038, avg 3µs/call # 427 times (933µs+0s) by Mail::SpamAssassin::Message::_parse_normal at line 1043, avg 2µs/call # 390 times (1.98ms+0s) by Mail::SpamAssassin::Message::_parse_multipart at line 984, avg 5µs/call # 198 times (1.10ms+0s) by Mail::SpamAssassin::Message::_parse_multipart at line 856, avg 6µs/call # 198 times (839µs+0s) by Mail::SpamAssassin::Message::_parse_multipart at line 848, avg 4µs/call # 171 times (576µs+0s) by Mail::SpamAssassin::Message::new at line 374, avg 3µs/call # 88 times (342µs+0s) by Mail::SpamAssassin::Message::_parse_multipart at line 1011, avg 4µs/call sub Mail::SpamAssassin::Message::CORE:match; # opcode
					# spent 4.42ms within Mail::SpamAssassin::Message::CORE:qr which was called 705 times, avg 6µs/call: # 705 times (4.42ms+0s) by Mail::SpamAssassin::Message::get_body_text_array_common at line 1111, avg 6µs/call sub Mail::SpamAssassin::Message::CORE:qr; # opcode
					# spent 32.5ms within Mail::SpamAssassin::Message::CORE:regcomp which was called 1253 times, avg 26µs/call: # 494 times (9.11ms+0s) by Mail::SpamAssassin::Message::_parse_multipart at line 884, avg 18µs/call # 390 times (6.70ms+0s) by Mail::SpamAssassin::Message::_parse_multipart at line 937, avg 17µs/call # 198 times (7.78ms+0s) by Mail::SpamAssassin::Message::_parse_multipart at line 848, avg 39µs/call # 171 times (8.91ms+0s) by Mail::SpamAssassin::Message::new at line 374, avg 52µs/call sub Mail::SpamAssassin::Message::CORE:regcomp; # opcode
					# spent 1.03ms within Mail::SpamAssassin::Message::CORE:sort which was called 235 times, avg 4µs/call: # 235 times (1.03ms+0s) by Mail::SpamAssassin::Message::get_all_metadata at line 584, avg 4µs/call sub Mail::SpamAssassin::Message::CORE:sort; # opcode
					# spent 49.8ms within Mail::SpamAssassin::Message::CORE:subst which was called 8145 times, avg 6µs/call: # 7440 times (20.7ms+0s) by Mail::SpamAssassin::Message::new at line 272, avg 3µs/call # 705 times (29.2ms+0s) by Mail::SpamAssassin::Message::get_body_text_array_common at line 1144, avg 41µs/call sub Mail::SpamAssassin::Message::CORE:subst; # opcode
					# spent 3.26ms within Mail::SpamAssassin::Message::CORE:unlink which was called 16 times, avg 203µs/call: # 16 times (3.26ms+0s) by Mail::SpamAssassin::Message::finish at line 667, avg 203µs/call sub Mail::SpamAssassin::Message::CORE:unlink; # opcode