GraphicsMagick 1.3.13之前版本的SyncBlob方法的一个BUG

BUG描述:

GraphicsMagick在处理多帧(multi-frames)图片(如GIF动画)时SyncBlob方法会修改后续帧的blob指针的指向:全部指向第一帧的blob!

如果后续处理过程中又调用到了SyncNextImageInList方法,继而SyncNextImageInList在调用ReferenceBlob的时候就会抛出下面的异常,从而导致进程退出:

server: ../GraphicsMagick-1.3.12/magick/semaphore.c:526: LockSemaphoreInfo: Assertion `semaphore_info->signature == 0xabacadabUL' failed.

注意:直接调用GraphicsMagick的gm命令处理处理多帧gif不会遇到上面的LockSemaphore异常;异常是发生在当你调用GraphicsMagick的API处理图片并试图获取图片的Blob数据的情况。

GraphicsMagick 1.3.12版本中的SyncBlob方法:

/*
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                                                                             %
%                                                                             %
%                                                                             %
+  S y n c B l o b                                                            %
%                                                                             %
%                                                                             %
%                                                                             %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%  SyncBlob() flushes the datastream if it is a file or synchonizes the data
%  attributes if it is an blob.
%
%  The format of the SyncBlob method is:
%
%      int SyncBlob(Image *image)
%
%  A description of each parameter follows:
%
%    o status:  Method SyncBlob returns 0 on success; otherwise,  it
%      returns -1 and set errno to indicate the error.
%
%    o image: The image.
%
%
*/
static int SyncBlob(Image *image)
{
  int
    status;

  register Image
    *p;

  assert(image != (Image *) NULL);
  assert(image->signature == MagickSignature);
  assert(image->blob != (BlobInfo *) NULL);
  assert(image->blob->type != UndefinedStream);
  for (p=image; p->previous != (Image *) NULL; p=p->previous);
  // 以下3行代码修改了后续帧的blob指针的指向:全部指向第一帧的blob
  for ( ; p->next != (Image *) NULL; p=p->next)
    if (p->blob != image->blob)
      *p->blob=(*image->blob);
  status=0;
  switch (image->blob->type)
  {
    case UndefinedStream:
      break;
    case FileStream:
    case StandardStream:
    case PipeStream:
    {
      status=fflush(image->blob->file);
      break;
    }
    case ZipStream:
    {
#if defined(HasZLIB)
      status=gzflush(image->blob->file,Z_SYNC_FLUSH);
#endif
      break;
    }
    case BZipStream:
    {
#if defined(HasBZLIB)
      status=BZ2_bzflush(image->blob->file);
#endif
      break;
    }
    case BlobStream:
      break;
  }
  return(status);
}

修复方法:
1) 升级GraphicsMagick到更新的版本(2011-12-24 GraphicsMagick发布了1.3.13版本)
2) 手工剔除SyncBlob方法中的第4049~4052行(未经测试!)

利用Unsharp Mask(USM)提高缩略图的质量(清晰度)

Unsharp masking的介绍

Unsharp Mask(USM),有人翻译为“非锐化遮罩”,也有人翻译为“虚化掩模锐化”,即在进行锐化计算的过程中,可对边缘进行锐化处理,而对色调连续的部分予以保护,达到既进行锐化,又不产生噪点的目的。

Unsharp Mask(USM)处理图片的实例:

详见:Sharpening: Unsharp Mask

Unsharp Mask Example Image
Original Unsharp Mask Sharpened

(Unsharp mask brightened slightly to increase visibility)

Unsharp Mask参数设置

详见:Unsharp Mask: suggested starting values by Simon Mackie

Scott Kelby suggests using the following settings as starting points for Unsharp Mask (USM) in Photoshop. Give them a go:

Subject Amount Radius Threshold
Soft subjects 150 1 10
Portraits 75 2 3
Moderate sharpening 225 0.5 0
Maximum sharpening 65 4 3
All-purpose sharpening 85 1 4
preparing for Web 400 0.3 0

Remember that these are just guidelines – oversharpened images always look terrible. Try these as a starting point, but if you get symptoms of oversharpening (grain in the image, and “halos” around objects), back off a little bit. And always check the image at at least 100% zoom.

GraphicsMagick & Unsharp Mask

$ /usr/local/GraphicsMagick/bin/gm convert -quality 85 -unsharp 0.5×1.0+0.8+0.03 original.jpg usm.jpg

-unsharp radius
-unsharp radiusxsigma{+amount}{+threshold}

sharpen the image with an unsharp mask operator.

The -unsharp option sharpens an image. The image is convolved with a Gaussian operator of the given radius and standard deviation (sigma). For reasonable results, radius should be larger than sigma. Use a radius of 0 to have the method select a suitable radius.

The parameters are:

   radius     The radius of the Gaussian, in pixels,  not counting the center
              pixel (default 0).
   sigma      The standard deviation of the Gaussian, in pixels (default 1.0).
   amount     The fraction of the difference between the original and the blur
              image that is added back into the original (default 1.0).
   threshold  The threshold, as a fraction of QuantumRange, needed to apply the
              difference amount (default 0.05).

Unsharp Mask for PHP

后记

针对小尺寸的缩略图(120×120以下),测试了几种Unsharp Mask的参数组合,最后觉得还是Torstein Hønsi总结的效果更好一些:

Amount: 80 (typically 50 – 200)
Radius: 0.5 (typically 0.5 – 1)
Threshold: 3 (typically 0 – 5)

附:UnsharpMask.php

Resin的 ‘Invocation miss ratio’比较高的情况下可以适当调小invocation-cache-size

invocation-cache-size 的缺省值是64*1024,即64K,于是LruCache$CacheItem的capacity是64*1024*2,即128K,如果每个LruCache$CacheItem占用2KB的空间的话,那么这么大的LruCache最多会占用128K*2KB,也就是256MB的Heap空间

  public LruCache(int initialCapacity, boolean isStatistics)
  {
    int capacity;

    for (capacity = 16; capacity < 2 * initialCapacity; capacity *= 2) {
    }

    _entries = new CacheItem[capacity];
    _prime = Primes.getBiggestPrime(_entries.length);

    _locks = new Object[(_entries.length >> 3) + 1];
    for (int i = 0; i < _locks.length; i++) {
      _locks[i] = new Object();
    }

    _capacity = initialCapacity;
    _capacity1 = _capacity / 2;

    if (_capacity > 32)
      _lruTimeout = _capacity / 32;
    else
      _lruTimeout = 1;

    _isEnableStatistics = isStatistics;

    /*
    if (isStatistics) {
      _hitCount = new AtomicLong();
      _missCount = new AtomicLong();
    }
    */
  }

注意观察Resin的 ‘Invocation miss ratio’,如果这个值比较高的话,说明invocationCache的命中率很低,这个时候可以适当调小invocation-cache-size参数:

  <cluster id="app-tier">
    <!-- sets the content root for the cluster, relative to resin.root -->
    <root-directory>.</root-directory>

    <invocation-cache-size>32</invocation-cache-size>

利用JMAP+MAT分析Java Heap Dump

如果你的Heap Dump文件超过了几百MB,那就不要再寄希望于jhat了,因为jhat需要数倍于dump文件的内存。这个时候你可以用MAT(Memory Analyzer),用MAT你可以在有2GB可用内存的机器上分析大约1GB左右的Dump文件。

生成Heap Dump文件的方法:

JMAP(Java Memory Map)

方法一:让运行中的JVM生成Dump文件
/usr/java/jdk/bin/jmap -F -dump:format=b,file=/path/to/heap/dump/heap.bin PID

方法二:让JVM在遇到OOM(OutOfMemoryError)时生成Dump文件
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/heap/dump

用jhat分析Dump文件
注:只有在Dump文件比较小的时候才适合用jhat分析Dump文件
jhat(Java Heap Analysis Tool)
jhat -stack false -refs false -J-Xmx2g /path/to/heap/dump/heap.bin

用MAT分析Dump文件
MAT(Memory Analyzer)

http://www.eclipse.org/mat/

安装MAT插件

在SunOS系统安装cvs客户端

根据系统的类型选择对应的发行版

$ uname -a
SunOS xxxxxx 5.10 Generic_141445-09 i86pc i386 i86pc

CVS

http://ftp.gnu.org/non-gnu/cvs/binary/stable/

$ mkdir bin
$ cd bin
$ wget http://ftp.gnu.org/non-gnu/cvs/binary/stable/x86-sunos/cvs-1.11.21-SunOS-5.8-i386.gz
$ gunzip cvs-1.11.21-SunOS-5.8-i386.gz
$ ln -s cvs-1.11.21-SunOS-5.8-i386 cvs

高负载下mod_caucho模块偶尔返回503错误

环境:Resin-2.1.17 + Apache-2.2.4,Resin编译为apache的一个module

现象:一个请求,apache日志里是503,而resin的日志里是200

apache的日志片段
219.239.xx.xxx – - [13/May/2011:09:15:06 +0800] “GET /newsreload?act=reload&path=/www/indices/news-test-all/index/sameNewsNum.txt HTTP/1.0″ 503 323 “-” “Wget/1.10.2 (Red Hat modified)”
resin的日志片段
219.239.xx.xxx - – [13/May/2011:09:15:08 +0800] “GET /newsreload?act=reload&path=/www/indices/news-test-all/index/sameNewsNum.txt HTTP/1.0″ 200 0 “-” “Wget/1.10.2 (Red Hat modified)”

分析mod_caucho的代码:

mod_caucho.c

/**
 * Handle a request.
 */
static int
caucho_request(request_rec *r)
{
   ...
   reuse = write_request(&s, r, config, &keepalive, session_index, backup_index);
   ...
  if (reuse == CSE_END || reuse == CSE_CLOSE)
    return OK;
  else
    return HTTP_SERVICE_UNAVAILABLE;
}

/**
 * handles a client request
 */
static int
write_request(stream_t *s, request_rec *r, config_t *config,
              int *keepalive, int session_index, int backup_index)
{
   ...
  code = send_data(s, r, CSE_END, keepalive);
  write_length = s->write_length;

  return code;
}

/**
 * Copy data from the JVM to the browser.
 */
static int
send_data(stream_t *s, request_rec *r, int ack, int *keepalive)
{
   ...
    switch (code) {
   ...
    case CSE_DATA:
      if (cse_write_response(s, len, r) < 0)
        return -1;
   ...
}

/**
 * Writes a response from srun to the client
 */
static int
cse_write_response(stream_t *s, int len, request_rec *r)
{
  while (len > 0) {
    int sublen;
    int writelen;
    int sentlen;

    if (s->read_offset >= s->read_length && cse_fill_buffer(s) < 0)
      return -1;

    sublen = s->read_length - s->read_offset;
    if (len < sublen)
      sublen = len;

    writelen = sublen;
    while (writelen > 0) {
      sentlen = ap_rwrite(s->read_buf + s->read_offset, writelen, r);
      if (sentlen < 0) {
        cse_close(s, "write");
        return -1;
      }

      writelen -= sublen;
    }

    s->read_offset += sublen;
    len -= sublen;
  }

  return 1;
}

参见:http://www.google.com/search?q=mod_caucho+503

GraphicsMagick处理图片Profile信息过程中一处内存泄露的BUG

In method AppendImageProfile(Image *image, const char *name, const unsigned char *profile_chunk, const size_t chunk_length), the pointer ‘unsigned char *profile’ was allocated via MagickAllocateMemory(unsigned char *,(size_t) profile_length) and it’s clone was added onto the image->profiles in method SetImageProfile(), the pointer itself remains, which should be freed.

/*
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                                                                             %
%                                                                             %
%                                                                             %
%     A p p e n d I m a g e P r o f i l e                                     %
%                                                                             %
%                                                                             %
%                                                                             %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%
%  AppendImageProfile adds a named profile to the image. If a profile with the
%  same name already exists, then the new profile data is appended to the
%  existing profile. If a null profile address is supplied, then an existing
%  profile is removed. The profile is copied into the image. Note that this
%  function does not execute CMS color profiles. Any existing CMS color
%  profile is simply added/updated. Use the ProfileImage() function in order
%  to execute a CMS color profile.
%
%  The format of the AppendImageProfile method is:
%
%      MaickPassFail AppendImageProfile(Image *image,const char *name,
%                                       const unsigned char *profile_chunk,
%                                       const size_t chunk_length)
%
%  A description of each parameter follows:
%
%    o image: The image.
%
%    o name: Profile name. Valid names are "8BIM", "ICM", "IPTC", XMP, or any
%                          unique text string.
%
%    o profile_chunk: Address of profile chunk to add or append. Pass zero
%               to remove an existing profile.
%
%    o length: The length of the profile chunk to add or append.
%
*/
MagickExport MagickPassFail
AppendImageProfile(Image *image,
               const char *name,
               const unsigned char *profile_chunk,
               const size_t chunk_length)
{
  const unsigned char
    *existing_profile;

  size_t
    existing_length;

  MagickPassFail
    status;

  status=MagickFail;
  existing_length=0;
  existing_profile=(const unsigned char *) NULL;
  if (profile_chunk != (const unsigned char *) NULL)
    existing_profile=GetImageProfile(image,name,&existing_length);

  if ((profile_chunk == (const unsigned char *) NULL) ||
      (existing_profile == (const unsigned char *) NULL))
    {
      status=SetImageProfile(image,name,profile_chunk,chunk_length);
    }
  else
    {
      unsigned char
      *profile;

      size_t
      profile_length;

      profile_length=existing_length+chunk_length;
      if ((profile_length < existing_length) ||
        ((profile=MagickAllocateMemory(unsigned char *,(size_t) profile_length)) ==
         (unsigned char *) NULL))
      ThrowBinaryException(ResourceLimitError,MemoryAllocationFailed,
                       (char *) NULL);
      (void) memcpy(profile,existing_profile,existing_length);
      (void) memcpy(profile+existing_length,profile_chunk,chunk_length);
      status=SetImageProfile(image,name,profile,profile_length);
    }

  return status;
}

The following is the patch:

--- GraphicsMagick-1.3.12/magick/profile.c 2009-12-17 02:49:54.000000000 +0800
+++ /opt/mod_imageman/GraphicsMagick-1.3.12/magick/profile.c 2011-04-29 00:44:57.000000000 +0800
@@ -176,6 +176,7 @@
(void) memcpy(profile,existing_profile,existing_length);
(void) memcpy(profile+existing_length,profile_chunk,chunk_length);
status=SetImageProfile(image,name,profile,profile_length);
+ MagickFreeMemory(profile);
}

return status;

补丁程序:http://sourceforge.net/tracker/?func=detail&aid=3294496&group_id=73485&atid=537939

GraphicsMagick无法正常渲染点阵字体的一处BUG及补丁

GraphicsMagick的annotate.c

By learning the code of annotate.c, gd library(gdft.c) and freetype2, I learned:

  1. the fixed-size bitmap font dose not support transform(see FT_Glyph_Transform), so the bitmap->left is always 0;
  2. the data structure of 1-bit monochrome bitmap’s bitmap.buffer differ from 8-bit gray-level pixmap, so the buffer should not be iterated with the same method.

补丁地址:http://sourceforge.net/tracker/?func=detail&aid=3230719&group_id=73485&atid=537939

/usr/local/GraphicsMagick/bin/gm convert -debug all -quality 90 -antialias -font “@/usr/share/fonts/bitmap-fonts/console8×8.pcf” -encoding UTF-8 -pointsize 8 -draw “text 20,40 ‘9876543210′” blank.jpg annotate_mess.jpg

补丁前:

补丁后:

Failed to compile resin-2.1.17 with Apache-2.2.16 while missing the ‘large file support’ CFLAGS

尝试把resin-2.1.17以apache2的module的方式编译,编译过程中遇到以下error:
/usr/local/apache2/include/apr.h:285:error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘apr_off_t’

缺省的编译选项

CFLAGS=-g -O2
make[2]: Entering directory `/usr/local/resin-2.1.17/src/c/plugin/apache2′
gcc -c -I/usr/local/apache2/include -DRESIN_HOME=\”/usr/local/resin-2.1.17\” -I../common -g -O2 -fpic mod_caucho.c && mv mod_caucho.o mod_caucho.lo
In file included from /usr/local/apache2/include/ap_config.h:25,
from /usr/local/apache2/include/httpd.h:43,
from mod_caucho.c:42:
/usr/local/apache2/include/apr.h:285:error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘apr_off_t’
In file included from /usr/local/apache2/include/apr_file_io.h:29,
from /usr/local/apache2/include/apr_network_io.h:26,
from /usr/local/apache2/include/httpd.h:53,
from mod_caucho.c:42:
apr.h文件中对应位置的声明
/usr/local/apache2/include/apr.h
284: typedef  ssize_t         apr_ssize_t;
285: typedef  off64_t           apr_off_t;
286: typedef  socklen_t       apr_socklen_t;
编译失败的原因:没有正确设置‘支持大文件’所需的编译选项
提取编译apr程序所需的cppflags:
$ /usr/local/apache2/bin/apr-1-config –cppflags
-DLINUX=2 -D_REENTRANT -D_GNU_SOURCE -D_LARGEFILE64_SOURCE
修改编译选项:
$ vi src/c/plugin/apache2/Makefile
CFLAGS=-g -O2 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DLINUX=2 -D_REENTRANT -D_GNU_SOURCE -D_LARGEFILE64_SOURCE
make[2]: Entering directory `/usr/local/resin-2.1.17/src/c/plugin/apache2′
gcc -c -I/usr/local/apache2/include -DRESIN_HOME=\”/usr/local/resin-2.1.17\” -I../common -g -O2 -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -DLINUX=2 -D_REENTRANT -D_GNU_SOURCE -D_LARGEFILE64_SOURCE -fpic mod_caucho.c && mv mod_caucho.o mod_caucho.lo
mod_caucho.c:926: warning: initialization from incompatible pointer type
mod_caucho.c:929: warning: initialization from incompatible pointer type
mod_caucho.c:932: warning: initialization from incompatible pointer type
mod_caucho.c:935: warning: initialization from incompatible pointer type
gcc -o mod_caucho.so -shared mod_caucho.lo ../common/stream.lo ../common/registry.lo ../common/config.lo ../common/memory.lo
make[2]: Leaving directory `/usr/local/resin-2.1.17/src/c/plugin/apache2′

编译成功

Foxmail .ind文件(邮箱索引文件)损坏后的修复方法

下午处理同事发来的邮件,打开附件中的excel文件时,excel提示”#$%^&*…”,然后Foxmail进程持续占用CPU 100%。无奈之下在Process Explorer把Foxmail Kill掉,但没想到的是重启Foxmail后Foxmail提示”*&^%$#@!…”,到Foxmail邮件的归档目录下,发现AAAAA48.IND文件变成了0个字节!

解决方法:

  1. 关闭Foxmail进程
  2. 复制另一个邮箱的.ind文件(.ind文件是邮箱的索引文件)为’AAAAA48.IND’
  3. 重启Foxmail
  4. 在出问题的邮件夹上点击右键,然后选择”属性>>邮件夹>>修复工具”,如下图:
    Foxmail邮件夹修复工具

    Foxmail邮件夹修复工具

    修复工具将尝试为邮件夹重建索引(遍历.box文件中的所有Entry,提取索引信息生成.ind文件),重建过程结束后出问题的邮件夹就可以正常使用了。

参见官方的帮助:

附”.box文件(邮箱数据文件)损坏后的修复方法”:

如果出错的是邮箱数据文件(.box文件),并且希望修复邮箱中的邮件,可以从以下方面尝试修复:如果文件被其他程序锁住了,可以退出锁住该文件的程序(例如杀毒软件程序),或者重新启动计算机。

如果找不到所提示的文件,说明文件丢失了,请检查邮箱文件是否被转移到其他位置,或者被杀毒软件隔离起来了。如果找到丢失的文件,请把文件复制回来。

如果文件被破坏了,可以尝试用写字板或者UltraEdit等文本编软件打开box文件,如果文件没有遭到彻底破坏,将可以看到部分邮件的原始信息。把已经被破坏的数据删除,保存文件,然后修复邮箱,将可以修复部分邮件。